ATF - About

About

The Automated Testing Framework (ATF) is a collection of libraries and utilities designed to ease unattended application testing in the hands of developers and end users of a specific piece of software.

As regards developers, ATF provides the necessary means to easily create test suites composed of multiple test programs, which in turn are a collection of test cases. It also attempts to simplify the debugging of problems when these test cases detect an error by providing as much information as possible about the failure.

As regards users, it simplifies the process of running the test suites and, in special, encourages end users to run them often: they do not need to have source trees around nor any other development tools installed to be able to certify that a given piece of software works on their machine as advertised.

Features

Automated testing: As ATF's name implies, its main feature is the execution of software tests in an automated fashion. This does not mean that tests as not flexible, though, as could be thought of processes that run without any user interaction. Testing can be configured prior to execution to adapt it to the system it will run under. Automating tests is very important to be able to run them frequently during the development of an application and to allow the end user to run them painlessly.
Installation of tests: There are many applications out there that provide test suites as part of their source code. These tests are usually run after the program is compiled, but never again. What is worse, these tests are typically executed on the machine where the software was built on instead of on the machine were the software will be run. This means that the results obtained by such execution do not necessarily match reality, as different installations of a system can differ in many areas causing incompatibilities and/or errors.

Therefore, one of ATF's key features is that test suites are installed on the system as part of the application's regular installation procedure. This means that these are always available on the machine were the program will be run and are easily reproducible by the end user. This can help a lot when debugging a problem exposed on one machine but not in any other, as it is very likely that some parts of the test suite will be affected by it.
Ease of use: More important than writing automated tests for a piece of software is the ability to run them all easily so that the whole test suite can be stressed periodically to ensure that the application remains functional. ATF makes it very easy to automatically run test suites, including the possibility to run many different ones at once. The results of these test suites are collected into nicely-formatted reports for later inspection by either the developer, the system administrator or any user.
Flexibility: ATF provides a framework to easily implement and run tests, but it imposes few restrictions on what the tests themselves do. This means that you can easily develop unit tests, regression tests, stress tests and any other kind of verification routine you need for your application.
Portability: ATF works on most Unix-like operating systems as well as on many different hardware platforms. The main development one is NetBSD/i386, but releases are fully tested under multiple different operating systems; each release announcement is accompanied by the list of platforms it was verified to work under. All that is needed is to get the framework to function is a C++ compiler and a POSIX shell interpreter.

Native Win32 support is not planned in the short term.
Modularity: ATF is very modular. It is composed of many small utilities that follow the traditional Unix design principle: they do one thing, but do that one right.

This design also permits test cases and test programs to be written in any language: they only have to conform to a plain-text I/O protocol defined by the framework to be usable inside ATF. ATF currently provides bindings to ease the implementation of tests in C/C++ and POSIX shell, but other bindings are planned. Among them, Python and Haskell are the ones that will probably come first.

Overview

For maximum flexibility, ATF test suites are organized as a tree of tests. These are as follows:

Test cases: The most basic testing unit. These do one specific test (or several highly-related ones) and return one of the following exit status: passed, skipped or failed.
Test programs: Test programs group a collection of test cases. They are completely self-contained and directly executable by the user.
Test suites: A collection of related test programs.

The developer has to develop all the above blocks to implement his test suite, but the user does not have to deal with the concepts above too much. He simply uses the execution engine (the atf-run utility) to automatically execute test suites or test programs and then uses the reporting engine (the atf-report utility) to transform the results into a nicely-formatted report. This is depicted below:

License

ATF is distributed under the terms of the very permissive, open-source TNF license. Its text is reproduced below:

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

History

ATF started as a Google Summer of Code 2007 project mentored by The NetBSD Foundation. Its original goal was to provide a testing framework for The NetBSD Operating System, but it grew as an independent project because the framework itself did not need to be tied to a specific operating system.

Alternatives

The following list describes alternative test frameworks and outlines how they differ from ATF. The basic descriptions of each tool is copied from the tool's site if available.

Autotest: Autotest is a collection of M4 macros, shipped as part of GNU Autoconf, whose purpose is to ease the creation of test programs in the POSIX shell language.

Autotest is not really a framework, as it is only useful to write test programs. It is also heavily coupled with the other Autotools. And, due to its M4-based design, the test programs generated by it tend to be huge (in the order of megabytes for not-so-big test suites). This is why, for example, Monotone replaced its Autotest-based test suite with a home-grown framework.

Some of the functions provided by ATF's POSIX shell binding where inspired by the testing macros found in Autotest.
Boost.Test: The Boost Test Library provides a matched set of components for writing test programs, organizing tests in to simple test cases and test suites, and controlling their runtime execution. It is typically used alongside Boost.Build, a make-like tool that has the necessary logic to build and optionally run a given set of tests.

This cannot be really considered a framework because Boost.Test only provides a library to simplify the creation of test programs. The framework appears when you join these test programs with some Boost.Build scripts. But these two components can add a lot of overhead in your project, and Boost.Build is often criticised for being hard to learn.

Some of the macros provided by ATF's C++ binding where inspired by the testing macros found in Boost.Test.
DejaGnu: A framework for testing other programs. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A test harness is the testing infrastructure that is created to support a specific program or tool. Each program can have multiple testsuites, all supported by a single test harness. DejaGnu is written in Expect, which in turn uses Tcl.

As is clear from the description, this framework relies on two non-standard tools: Expect and Tcl. Developers do not usually know these languages, and adding these as a dependencies for tests might be considered bad. Regardless, DejaGnu is used intensively by many important GNU components, such as GCC or Binutils, which shows its maturity.

It is also worth to note that none of the frameworks shown above consider the tests as something that the end user should run periodically. The tests constructed by them are shipped as part of a source package and are only run after the program is built. The end user, who almost always uses binary packages, never gets to see these tests nor run them on his own machine (with its specific hardware, software and operating system combination). ATF, on the other hand, does, and this is probably the main difference between it and all other available frameworks.