BTest - A Simple Driver for Basic Unit Tests

The btest is simple framework for writing unit tests. Freely borrowing some ideas from other packages, it’s main objective to provide am easy-to-use, straight-forward driver for a suite of shell-based tests. Each test consists of a set of command lines that will be executed, and success is determined based on their exit codes. btest comes with some additional tools that can be used within such tests to compare output against a previously established baseline.

Contents

BTest - A Simple Driver for Basic Unit Tests

Download

You can find the latest BTest release for download at http://www.bro-ids.org/download.

BTest’s git repository is located at git://git.bro-ids.org/btest.git. You can browse the repository here.

This document describes BTest 0.31-23. See the CHANGES file for version history.

Installation

Installation is simple and standard:

tar xzvf btest-*.tar.gz
cd btest-*
python setup.py install

This will install a few scripts: btest is the main driver program, and there are number of further helper scripts that we discuss below (including btest-diff, which is a tool for comparing output to a previously established baseline).

Writing a Simple Test

In the most simple case, btest simply executes a set of command lines, each of which must be prefixed with @TEST-EXEC::

> cat examples/t1
@TEST-EXEC: echo "Foo" | grep -q Foo
@TEST-EXEC: test -d .
> btest examples/t1
examples.t1 ... ok

The test passes as both command lines return success. If one of them didn’t, that would be reported:

> cat examples/t2
@TEST-EXEC: echo "Foo" | grep -q Foo
@TEST-EXEC: test -d DOESNOTEXIST
> btest examples/t2
examples.t2 ... failed

Usually you will run just all tests found in a directory:

> btest examples
examples.t1 ... ok
examples.t2 ... failed
1 test failed

Why do we need the @TEST-EXEC: prefixes? Because the file containing the test can simultaneously acts as its input. Let’s say we want to verify a shell script:

> cat examples/t3.sh
# @TEST-EXEC: sh %INPUT
ls /etc | grep -q passwd
> btest examples/t3.sh
examples.t3 ... ok

Here, btest is executing (something similar to) sh examples/t3.sh, and then checks the return value as usual. The example also shows that the @TEST-EXEC prefix can appear anywhere, in particular inside the comment section of another language.

Now, let’s say we want to check the output of a program, making sure that it matches what we expect. For that, we first add a command line to the test that produces the output we want to check, and then run btest-diff to make sure it matches a previously recorded baseline. btest-diff is itself just a script that returns success if the output is as expected, and failure otherwise. In the follwing example, we use an awk script as a fancy way to print all file names starting with a dot in the user’s home directory. We write that list into a file called dots and then check whether its content matches what we know from last time:

> cat examples/t4.awk
# @TEST-EXEC: ls -a $HOME | awk -f %INPUT >dots
# @TEST-EXEC: btest-diff dots
/\.*/ { print $1 }

Note that each test gets its own litte sandbox directory when run, so by creating a file like dots, you aren’t cluttering up anything.

The first time we run this test, we need to record a baseline:

> btest -U examples/t4.awk

Now, btest-diff has remembered what the dots file should look like:

> btest examples/t4.awk
examples.t4 ... ok
> touch ~/.NEWDOTFILE
> btest examples/t4.awk
examples.t4 ... failed
1 test failed

If we want to see what exactly the unexpected change is that was introduced to dots, there’s a diff mode for that:

> btest -d examples/t4.awk
examples.t4 ... failed
% 'btest-diff dots' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
[... current dots file ...]
== Diff ===============================
--- /Users/robin/work/binpacpp/btest/Baseline/examples.t4/dots
2010-10-28 20:11:11.000000000 -0700
+++ dots      2010-10-28 20:12:30.000000000 -0700
@@ -4,6 +4,7 @@
.CFUserTextEncoding
.DS_Store
.MacOSX
+.NEWDOTFILE
.Rhistory
.Trash
. Xauthority
=======================================

% cat .stderr
[... if any of the commands had printed something to stderr, that would follow here ...]

Once we delete the new file, we are fine again:

> rm ~/.NEWDOTFILE
> btest -d examples/t4.awk
examples.t4 ... ok

That’s already the main functionality that the btest package provides. In the following, we describe a number of further options extending/modifying this basic approach.

Reference

Command Line Usage

btest must be started with a list of tests and/or directories given on the command line. In the latter case, the default is recursively scan the directories and assume all files found to be tests to perform. It is however possible to exclude certain files by specifying a suitable configuration file.

btest returns exit code 0 if all tests have succesfully passed, and 1 otherwise.

btest accepts the following options:

-b, --brief Does not output anything for tests which pass. If all tests pass, there will not be any output at all.

-c CONFIG, --config=CONFIG

Specifies an alternative configuration file to use. If not speicfiied, the default is to use a file called btest.cfg if found in the current directory.

-d, --diagnostics

Reports diagnostics for all failed tests. The diagnostics include the command line that failed, its output to standard error, and potential additional information recorded by the command line for diagnostic purposes (see @TEST-EXEC below). In the case of btest-diff, the latter is the diff between baseline and actual output.

-D, --diagnostics-all

Reports diagnostics for all tests, including those which pass.

-f DIAGFILE, --file-diagnostics=DIAGFILE

Writes diagnostics for all failed tests into the given file. If the file already exists, any new output is appended.

-F FILTER, --filter=FILTER

Activates a filter defined in the configuration file.

-g GROUP, --group=GROUP

Runs only tests assigned to the given test group, see @TEST-GROUP. Specifying - as the group name runs all tests that do not belong to any group.

-j THREADS, --jobs=THREADS

Runs up to the given number of tests in parallel. By default, BTest assumes that all tests can be executed concurrently without further constraints. One can however ensure serialization of subsets by assigning them to the same test group, see @TEST-GROUP.

-r, --rerun Runs only tests that failed last time. After each execution, BTest generates a state file that records the tests that have failed. Using this option on the next run then reads that file back in and limits execution to those tests found in there.

-S SUBST, --subst=SUBST

Activates a substitution defined in the configuration file.

-t, --tmp-keep Does not delete any temporary files created for running the tests (including their outputs). By default, the temporary files for a test will be located in .tmp/<test>/, where <test> the path to the test file with all slashes replaced with dots (e.g., the files for example/t3.sh will be in .tmp/example.t3.sh.

-U, --update-baseline

Records a new baseline for all btest-diff commands found in any of the specified tests. To do this, all tests are run as normal except that when btest-diff is executed, is does not compute a diff but instead considers the given file to authoritative and records it as the version to compare with in future runs.

-u, --update-interactive

For failed tests, btest will stop and ask whether the user wants to record a new baseline.

-v, --verbose Shows all test command lines as they are executed.

-w, --wait Interactively waits for <enter> after showing diagnostics for a test.

Configuration

Specifics of btest‘s execution can be tuned with a configuration file, which by default is btest.cfg if that’s found in the current directory. It can alternatively be specificied with the --config command line option. The configuration file is “INI-style”, and an example comes with the distribution, see btest.cfg.example. A configuration file has one main section, btest, that defines most options; as well as an optional section for defining environment variables and further optional sections for defining custom filters.

Note that all paths specified in the configuration file are relative to btest‘s base directory. The base directory is either the one where the configuration file is located if such is given/found, or the current working directory if not. When setting values for configuration options, the absolute path to the base directory is available by using the macro %(testbase)s (the weird syntax is due to Python`’s ConfigParser module ...).

Furthermore, all values can use standard “backtick-syntax” to include the output of external commands (e.g., xyz=`echo test`). Note that the backtick expansion is performed after any %(..) have already been replaced (including within the backticks).

Options

The following options can be set in the btest section of the configuration file:

TestDirs: A space-separated list of directories to search for tests. If defined, one doesn’t need to specify any tests on the command line.
TmpDir: A directory where to create temporary files when running tests. By default, this is set to %(testbase)s/.tmp.
BaselineDir: A directory where to store the baseline files for btest-diff. By default, this is set to %(testbase)s/Baseline.
IgnoreDirs: A space-separated list of relative directory names to ignore when scanning test directories recursively. Default is empty.
IgnoreFiles: A space-separated lists of filename globs matching files to ignore when wscanning given test directories recursively. Default is empty.
StateFile: The name of the state file to record failing tests in. Default is .btest.failed.dat.
Finalizer: An executable that will be executed each time any test has succesfully run. It runs in the same directory as the test itself and receives the name of the test as its parameter. The return value indicates whether the test should indeed be considered succeeded. By default, there’s no finalizer set.

Environment Variables

A special section environment defines environment variables that will be propapated to all tests:

[environment]
CFLAGS=-O3
PATH=%(testbase)s/bin:%(default_path)s

Note how PATH can be adjusted to include local scripts: the example above prefixes it with a local bin/ directory inside base directory, using the predefined default_path macro to refer to the PATH as it is set by default.

Furthermore, by setting PATH to include the btest distribution directory, one could skip the installation of the btest package.

Filters

Filters are a transparent way to adapt the input to a specific test command before it is executed. A filter is defined by adding a section [filter-<name>] to the configuration file. This section must have exactly one entry, and the name of that entry is interpreted as the name of a command which’s input is to be filtered. The value of that entry is the name of a filter script that will be with two arguments representing input and output files, respectively. Example:

[filter-cat]
cat=%(testbase)/bin/filter-cat

Once the filter is activated by running btest with --filter=cat, every time a @TEST-EXEC: cat %INPUT is found, btest will first execute (something similar to) filter-cat %INPUT out.tmp, and then subsequently cat out.tmp (i.e., the original command but with the filtered output). In the simplest case, the filter could be a no-op in the form cp $1 $2.

Note

There are few limitations to the filter concept currently:

Filters are always fed with %INPUT as their first argument. We should add a way to filter other files as well.
Filtered commands are only recognized if they are directly starting the command line. For example, @TEST-EXEC ls | cat >outout would not trigger the example filter above.
Filters are only executed for @TEST-EXEC, not for @TEST-EXEC-FAIL.

Substitutions

Substitutions are similar to filters, yet they do net adapt the input but the command line being exectued. A substitutions is defined by adding a section [subst-<name>] to the configuration file. This section must have exactly one entry, and the name of that entry is interpreted as the name of a command that is to be replaced with something else. Example:

[subst-opt]
gcc=gcc -O2

Once the substitution is activated by running btest with --subst=opt, every time a @TEST-EXEC executes gcc, that is replaced with gcc -O2. The replacement is simple string substitution so it works in fact not only with commands but anything found in the command line; it however only replaces full words, not subparts of words.

Writing Tests

btest scans a test file for lines containing keywords that trigger certain functionality. Currently, the following keywords are supported:

@TEST-EXEC: <cmdline>

Executes the given command line and aborts the test if it returns an error code other than zero. The <cmdline> is passed to the shell and thus can be a pipeline, use redirection, etc.

When running a test, the current working directory for all command lines will be set to a temporary sandbox (and later be deleted).

There are two macros that can be used in <cmdline>: %INPUT will be replaced with the name of the file defining the test; and %DIR will be replaced with the directory where the test file is located. The latter can be used to reference further files also located there.

In addition to environment variables defined in the configuration files, there are further ones that are passed into the commands:

TEST_DIAGNOSTICS

A file where further diagnostic information can be saved in case a command fails. --diagnostics will show this file. (This is also where btest-diff stores its diff.)

TEST_MODE

This is normally set to TEST, but will be UPDATE if btest is run with --update-baseline and UPDATE_INTERACTIVE with --update-interactive.

TEST_BASELINE

A filename where the command can save permanent information across btest runs. (This is where btest-diff stores its baseline in UPDATE mode.)

TEST_NAME

The name of the currently executing test.

Note

If a command returns the special exit code 100, the test is considered failed, however subsequent test commands are still run. btest-diff uses this to continue in the case that no baseline has yet been established.

If a command returns the special exit code 200, the test is considered failed and all further test executions are aborted.

@TEST-EXEC-FAIL: <cmdline>

Like @TEST-EXEC, except that this expects the command to fail, i.e., the test is aborted when the return code is zero.

@TEST-REQUIRES: <cmdline>

Defines a condition that must be met for the test to be executed. The given command line will be run before any of the actual test commands, and it must return success for the test to continue. If it does not return success, the rest of the test will be skipped but doing so will not be considered an error. This allows to write conditional tests that may not always make sense to run, depending on whether external constraints are satisfied or not (say, whether a particular library is available). Multiple requirements may be specified and then all must be met for the test to continue.

@TEST-START-NEXT

This is a short-cut for defining multiple test inputs in the same file, all executing with the same command lines. When @TEST-START-NEXT is encountered, the test file is initially considered to end at that point, and all @TEST-EXEC-* are run with an %INPUT truncated accordingly. Afterwards, a new %INPUT is created with everything following the @TEST-START-NEXT marker, and the same commands are run again (further @TEST-EXEC-* will be ignored). The effect is that a single file can actually define two tests, and the btest output will enumerate them:

> cat examples/t5.sh
# @TEST-EXEC: cat %INPUT | wc -c >output
# @TEST-EXEC: btest-diff output

This is the first test input in this file.

# @TEST-START-NEXT

... and the second.

> ./btest -D examples/t5.sh
examples.t5 ... ok
  % cat .diag
  == File ===============================
  119
  [...]

examples.t5-2 ... ok
  % cat .diag
  == File ===============================
  22
  [...]

Multiple @TEST-START-NEXT can be used to create more than two tests per file.

@TEST-START-FILE <file>

This is used to include an additional input file for a test right inside the test file. All lines following the keyword will be written into the given file (and removed from the test’s %INPUT) until a terminating @TEST-END-FILE is found. Example:

> cat examples/t6.sh
# @TEST-EXEC: awk -f %INPUT <foo.dat >output
# @TEST-EXEC: btest-diff output

    { lines += 1; }
END { print lines; }

@TEST-START-FILE foo.dat
1
2
3
@TEST-END-FILE

> btest -D examples/t6.sh
examples.t6 ... ok
  % cat .diag
  == File ===============================
  3

Multiple such files can be defined within a single test.

Note that this is only one way to use further input files. Another is to store a file in the same directory as the test itself, making sure it’s ignored via IgnoreFiles, and then refer to it via %DIR/<name>.

@TEST-GROUP: <group>

Assigns the test to a group of name <group>. Groups have two purposes:

By using option -g one can limit execution to all tests that belong to certain group.

When using option -j to parallelize execution, all tests part of the same group are guaranteed to run sequentially.

Canonifying Diffs

btest-diff has the capability to filter its input through an additional script before it compares the current version with the baseline. This can be useful if certain elements in an output are expected to change (e.g., timestamps). The filter can then remove/replace these with something consistent. To enable such canonification, set the environment variable TEST_DIFF_CANONIFIER to a script reading the original version from stdin and writing the canonified version to stdout. Note that both baseline and current output are passed through the filter before their differences are computed.

Running Processes in the Background

Sometimes processes need to be spawned in the background for a test, in particular if multiple processes need to cooperate in some fashion. btest comes with two helper scripts to make life easier in such a situation:

btest-bg-run <tag> <cmdline>: This is a script that runs <cmdline> in the background, i.e., it’s like using cmdline & in a shell script. Test execution continues immediately with the next command. Note that the spawned command is not run in the current directory, but instead in a newly created sub-directory called <tag>. This allows spawning multiple instances of the same process without needing to wory about conflicting outputs. If you want to access a command’s output later, like with btest-diff, use <tag>/foo.log to access it.

btest-bg-wait [-k] <timeout>

This scripts waits for all processes previousluy spawned via btest-bg-run to finish. If any of them exits with a non-zero return code, btest-bg-wait does so as well, indicating a failed test. <timeout>> is mandatory and gives the maximum number of seconds to wait for any of the processes to terminate. If any process hasn’t done so when the timeout expires, it will be killed and the test is considered to be failed as long as -k is not given. If -k is given, pending processes are still killed but the test continues normally, i.e., non-termination is not considered a failure in this case. This script also collect the processes’ stdout and stderr outputs for diagnostics output.

To-Do List

The following features would be nice to have:

Parallelize execution of tests for faster execution on multi-core systems (prototype exists).
Output test results in a machine-readable format that can then be parsed and pretty-printed by some frontend tool (potentially assembling and aggregating output from different systems).
Add an auto-tester script that runs a test-suite automatically in regular periods, mailing out reports when tests fail. There is an initial version of such a script, but it’s currently project-specific and needs to be generalized.

License

btest is open-source under a BSD licence.