Automated Testing Framework (ATF) News

ATF 0.5 released

jmmv — Thu, 01 May 2008 17:00:00 GMT

I am very pleased to announce the availability of ATF's fifth public release, 0.5, which is probably the one with the most code changes so far. The main goal of this release is to provide a C-only binding to write test cases, completely freeing the developers of the use of C++. This new C library weights at around 7000 lines and the C++ library has lost around 1000 due to the fact that it now reuses code in the C library. Another important change of this release is the relicensing of the complete code base under a 2-clause BSD license.

The major changes in this release are:

Clauses 3 and 4 of the BSD license used by the project were dropped. All the code is now under a 2-clause BSD license compatible with the GNU General Public License (GPL). Read the new license for further details.
Added a C-only binding so that binary test programs do not need to be tied to C++ at all. This binding is now known as the atf-c library.
Renamed the C++ binding to atf-c++ for consistency with the new atf-c.
Renamed the POSIX shell binding to atf-sh for consistency with the new atf-c and atf-c++.
Added a -w flag to test programs through which it is possible to specify the work directory to be used. This was possible in prior releases by defining the workdir configuration variable (-v workdir=...), but was a conceptually incorrect mechanism.
Test programs now preserve the execution order of test cases when they are given in the command line. Even those mentioned more than once are executed multiple times to comply with the user's requests.

The checksums for atf-0.5.tar.gz are:

MD5: b1d078b716e344bcf65f735cd74dc5d8
SHA1: 63cb8541608518cdb214a10e71228d3e8c8af514
SHA512: f78cce89 bfa34810 9e4bbd5c 20ea1930 8f8e3c4a 835b22cc 60e0e300 d243f98a ab4180ac 28673767 f34bbddf 1ad36acc 7569f483 0773d391 c5a38df4 1d6883a2

Go to the downloads page to get it!

C-only library for test programs

jmmv — Mon, 18 Feb 2008 11:15:00 GMT

The branch org.NetBSD.atf.src.c has just hit the public repository. This contains a preliminary implementation of a C-only library for test programs. It will allow developers using ATF in their projects to restrict their test programs to the C language without having to use C++ at all, which is a highly desired feature in, for example, the NetBSD source tree.

The main focus of 0.5 will be the stabilization of this branch and its integration into mainline, assuming there is no important bug fix to roll off before this happens.

See some more details in this atf-devel post. Feel free to raise your comments there.

ATF 0.4 released

jmmv — Thu, 04 Feb 2008 21:00:00 GMT

I am very pleased to announce the availability of ATF's fourth public release, 0.4. The initial main goal of this release was to provide some developer-specific documentation, but once I attacked the implementation of the features planned in the roadmap, things got more complex. In the end, this release brings several new features and lots of portability fixes.

The major changes in this release are:

Added two new manual pages, atf-c++-api and atf-sh-api, describing the C++ and POSIX shell interfaces used to write test programs.
Added a pkg-config file, useful to get the flags to build against the C++ library or to easily detect the presence of ATF.
Added a way for test cases to require a specific architecture and/or machine type through the new require.arch and require.machine meta-data properties, respectively.
Added the timeout property to test cases, useful to set an upper-bound limit for the test's run time and thus prevent global test program stalls due to the test case's misbehavior.
Added the atf-exec(1) internal utility, used to execute a command after changing the process group it belongs to.
Added the atf-killpg(1) internal utility, used to kill process groups.
Multiple portability fixes. Of special interest, full support for SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 C++ compiler.
Fixed a serious bug that prevented atf-run(1) from working at all under Fedora 8 x86_64. Due to the nature of the bug, other platforms were likely affected too.

The checksums for atf-0.4.tar.gz are:

MD5: df5b8794577cf962ccb7a0434644c9a3
SHA1: dce4885a2f2d55d73d5f248fe482d5ac3101ce86
SHA512: 93bb92bb 50784092 5ba10613 240360e6 eef605c9 e1eafff7 c0bcdc4b b27c27cd b272d479 cedf3bee b37d7550 bcb58ab5 c1fdb893 04492277 9bec29c8 4c9659ce

Go to the downloads page to get it!

ATF imported into NetBSD-current

jmmv — Thu, 12 Nov 2007 21:20:00 GMT

As I mentioned yesterday, ATF was hopefully ready to be integrated into NetBSD with the release of the 0.3 version. And it was! As there seem to be no serious problems with it, and as some people were adding pressure to get the thing integrated, I merged ATF into the NetBSD-current source tree today. Or, in other words, ATF will be part of NetBSD 5.0!

This marks a big milestone in the project, because ATF is now really put into use by an important and huge project: The NetBSD Operating System. This will hopefully boost interest in ATF and raise external contributions to either the core framework and/or to the NetBSD-specific tests.

For more details please refer to my official announcement at the tech-userlevel mailing list or to the news item at the NetBSD site.

ATF 0.3 released

jmmv — Thu, 11 Nov 2007 21:45:00 GMT

I am very pleased to announce the availability of ATF's third public release, 0.3. This release marks a big milestone for the ATF project: it is planned to be the first one to be integrated into the NetBSD source tree. Integration into the NetBSD operating system was the original goal of the ATF project as part of a Google Summer of Code 2007 project but, as it was not good enough when the program ended (or at least I considered it was not ready), integration was delayed. It is now polished enough to take make this important step forward!

The major changes in this release are:

Added XML output support to atf-report. This is accompanied by a DTD for the format's structure and sample XSLT/CSS files to post-process this output and convert it to a plain HTML report.
Changed atf-run to add system information to the report it generates. This is currently used by atf-report's XML output only, and is later printed in the HTML reports in a nice and useful summary table. The user and system administrator are allowed to tune this feature by means of hooks.
Removed the test cases' isolated property. This was intended to avoid touching the file system at all when running the related test case, but this has not been true for a long while: some control files are unconditionally required for several purposes, and we cannot easily get rid of them. This way we remove several critical and delicate pieces of code.
Improved atf-report's CSV output format to include information about test programs too.
Fixed the tests that used atf-compile to not require this tool as a helper. Avoids systems without build-time utilities to skip many tests that could otherwise be run. (E.g. NetBSD without the comp.tgz set installed.)
Many general cleanups: Fixed many pieces of code marked as ugly and/or incomplete.

The checksums for atf-0.3.tar.gz are:

MD5: b01dde98a3950f4cbdb914197455dab7
SHA1: 3734ed79d6116e4ec3c90ca2cf5aca5bac86e5a1
SHA512: ae353c4a b1356bea 92dbbf9f 53a8c1fb e381fffb e4f09959 3580926d 242be393 25043592 440a64d1 b14be895 ebef8ded 05716c89 04cc309c a6031933 8fc3c7ba

Go to the downloads page to get it!

ATF 0.2 released

jmmv — Thu, 20 Sep 2007 23:15:00 GMT

Exactly a month ago, the Google Summer of Code 2007 program concluded and, coinciding with that date, I published ATF 0.1. Development has continued since then and I am now pleased to announce the availability of the ATF 0.2 release, the first release outside of the program; hopefully many more will come. This new version has still many rough edges but given that the goals planned for it have been accomplished, I felt it was time to make it public.

The major changes in this release are:

Test cases now get a known umask on entry.
atf-run now detects many unexpected failures caused by test programs and reports them as bogus tests. atf-report is able to handle these new errors and nicely reports them to the user.
All the data formats read and written by the tools have been documented and cleaned up. These include those grammars that define how the different components communicate with each other as well as the format of files written by the developers and users: the Atffiles and the configuration files.
Added the atf-version tool, a utility that displays information about the currently installed version of ATF.
Test cases can now define an optional cleanup routine to undo their actions regardless of their exit status.
atf-report now summarizes the list of failed (bogus) test programs when using the ticker output format.
Test programs now capture some termination signals and clean up any temporary files before exiting the program.
Multiple bug fixes and improvements all around.

For a list of supported platforms in this release please see the downloads page. From now on this information will be kept there.

The checksums for atf-0.2.tar.gz are:

MD5: 163331daef2a41caafda9f08dc7bf5cd
SHA1: f31cf24df78c194b63c0694c0f1a7b2f7bb0c9b6
SHA512: d6edbba3 15da21fc 9b8c6318 18c42aa1 0dc44d00 2af045bf 4bfd1e21 835912b1 5914b383 c84d32d6 48c4e421 10ed5049 460a3627 9016f14e e84e2905 37adb98a

Go to the downloads page to get it!

Second preview of NetBSD with ATF

jmmv — Mon, 20 Aug 2007 21:40:34 GMT

I have just updated the first preview of NetBSD-current release builds with ATF merged in to match the ATF 0.1 release published today. As already stated in the old news item: These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF is imported into it; we are not sure when that will happen, though.

By the way, ATF 0.1 is also in pkgsrc as devel/atf. This may be an easier way for you to try it, even though you will not get the NetBSD-specific tests.

For more details see my post to the NetBSD's current-users mailing list.

ATF 0.1 released

jmmv — Mon, 20 Aug 2007 13:01:00 GMT

I am very pleased to announce the first public release of ATF, numbered 0.1 as you may expect. This release is being published to coincide with the end of the Google Summer of Code 2007, a program that sponsored this initial work in the project. Development will continue from now on with the primary goal of integrating ATF into the NetBSD operating system. But, of course, feel free to adopt ATF in your own applications.

This release has been tested under NetBSD-current (4.99.29), NetBSD 4.0_BETA2, Mac OS X 10.4.10 and openSUSE 10.2.

The checksums for atf-0.1.tar.gz are:

MD5: e043545044fa3aa9da0dd2a62712e1b6
SHA1: 3f3b8355d30f1372a5e3e5fed8621441956b6a4d
SHA512: 5ae09aab a448cfe7 d01b807c e3159164 d6138c2c bc335b7b da537403 e6455e87 8ba737fe 72c2bbea a97d9f9f 75c9a0d1 d65ddf16 a3a66a47 cd84fba3 487e756d

Go to the downloads page to get it!

New mailing list: atf-devel

jmmv — Sat, 18 Aug 2007 16:32:12 GMT

In preparation for the first public release of ATF, which will be released on the 20th, the project has got its first mailing list addressed at covering development-related topics and, at the moment, user-specific questions too. Its name is atf-devel, and you can find more details about it in the mailing lists page.

The description for the list reads:

atf-devel is the mailing list to discuss development issues related to the Automated Testing Framework (ATF) and is not tied to any specific project using ATF. Given that ATF is still very young and that the list is low-traffic, this is also the appropriate place to ask user-related questions. Inquiries related to the integration of ATF into NetBSD should be directed to tech-kern or tech-userlevel instead.

Many thanks to S.P.Zeidler from the NetBSD administration team for setting this up so quickly.

First preview of NetBSD with ATF

jmmv — Wed, 8 Aug 2007 01:52:12 GMT

I have just uploaded some NetBSD-current release builds with ATF merged in. These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF 0.1 is made public, which should happen later this month.

For more details see my post to the NetBSD's current-users mailing list.

Web site launched

jmmv — Sun, 16 Jul 2007 08:00:00 GMT

The official web site for ATF was launched. The old page will live until the Google Summer of Code 2007 program is over. If you ever visited the web site of the now-dead Buildtool project, this page will look very familiar. I did not want to rewrite everything from scratch, so I just copied several files from there, modified them a bit and liked the results.

Oh, and by the way, ATF has also got a logo! See it in the top-left corner of the page. It is pretty simple and may be replaced if somebody else comes up with a better one. But it is cool-enough for now :-)

Code made public

jmmv — Tue, 10 Jul 2007 17:00:00 GMT

ATF's code has been made public just in time for the Summer of Code 2007 midterm evaluation. You can find more information on how to access these sources (in early stages of development) at the repository page.

ATF selected as a Google Summer of Code 2007 project

jmmv — Thu, 12 Apr 2007 10:00:00 GMT

NetBSD was in the need of a testing framework to replace its current home-grown tests. ATF was presented as a project proposal to resolve these needs, and Julio M. Merino Vidal (the one writing these lines :-) was selected to work on it. Stay tuned for further news.