Testing NetBSD automagically

ATF comes from the GSoC 2007. It has a successor already announced, kyua, which is almost ready to take over.

ATF has bindings in shell, C, and others languges. It is easily scriptable, allows fully automated runs as easy as make regress and produces nice reports (in ASCII, HTML, XML amongst others). It does full runs, without stopping at the first error. Tests are organised in a hierarchy, and it is possible to run only a part (e.g. only the kernel).

It currently runs several times a day (thus requiring a buildable CVS; more on this later).

Test programs can run outside of ATF, they can be pushed to users for them to run.

Rump enables one to run kernel code in user space. It consists of a set of shlibs, each representing a slice of the full kernel. Client code (mostly userland) is compiled like a new architecture. Rump operates on file system images (regular file for the host system). Rump make it easier to debug (userland VS kernel).

There is a librumphijack, LD_PRELOAD'able, for instance to give a new TCP stack to unmodified binaries.

QEMU is a different architecture, from the point of view of build.sh. It allows easy scripting, which is very important for the testing, plus access to the serial console of the guest.

It has some rough edges: some failures come from the emulated FPU notably.

A python script that uses a release or a freshly downloaded snapshot, creates a brand new QEMU environment and run the full test suite. In itself, the installation of the new release is already a good test.

Anita takes about 1h30 to run.

The test environment is just like good ol' /bin/sh, just with a set of shell functions and variables already defined.

A test describes its requirements (for instance, I need to have access to cc(1)). Then, it describes the command to be ran, and the exit code and the output expected. The tests will be run in a clean, empty directory.

The C interface to ATF is a series of pre-processor macros, describing meta-information about the program and allowing ATF to check the behaviour of the library under test.

This is more difficult to test. A new $TERM was defined, in which control sequences are displayed verbatim. Two programs are involved, a director (programmed in some DSL), and a slave; the two communicate over a socket. The time between keystrokes and between screen updated is handled. The diff with the golden output was designed to be easy to read.

Martin showed a test case in which a pair of TTYs are opened in the wrong order; this made the running kernel crash. This bug report was not easy to debug in kernel code; it was much easier to reproduce and debug in userlang code.

Martin demonstrated us a small setup with three Rump instances, one router in which tcpdump(8) was running and two endpoints.

In this setup, the NICs are implemented as shmem interfaces.