Posted by jmmv on Sat, 06 Mar 2010 19:16:00 GMT | Permanent link

Yesterday, I announced the org.NetBSD.atf.src.nofork branch, an experimental prototype to remove the forking and isolation of test cases from within test programs (don't worry, the isolation is now in atf-run). This feature has been highly requested because the previous design of test programs gets in the way when attempting to debug a failing test case. The new approach makes debugging easier and simplifies, a lot, the internal ATF code.

For more details, please refer to the announcement posted to the atf-devel mailing list. I hope this new code base will be the foundation of the 0.8 release.

Posted by jmmv on Tue, 22 Dec 2009 12:37:00 GMT | Permanent link

The long-awaited 0.7 release is finally here! It has been a very long time since 0.6 and this release does not carry some of the improvements that were expected. I am publishing this now to give exposure to the new internals and to free me from blocking changes, thus finally being able to work on new features for 0.8.

If you are wondering why it takes so long between releases, here is why: preparing a release is a very time-consuming process and it is hard to convince oneself that it is the moment to do so. If it wasn't for this, trust me, there would be more frequent releases because the mainstream code is rarely broken to the point of being unusable.

Without further ado, the major changes in this release are:

• Added build-time checks to atf-c and atf-c++. A binding for atf-sh will come later.

• Migrated all build-time checks for header files to proper ATF tests. This demonstrates the use of the new feature described above.

• Added an internal API for child process management.

• Converted all plain-text distribution documents to a Docbook canonical version, and include pre-generated plain text and HTML copies in the distribution file.

• Simplified the contents of the Makefile.am by regenerating it from a canonical Makefile.am.m4 source. As a side-effect, some dependency specifications were fixed.

• Migrated all checks from the check target to installcheck, as these require ATF to be installed.

• Fixed sign comparison mismatches triggered by the now-enabled -Wsign-compare.

• Fixed many memory and object leaks.

The checksums for atf-0.7.tar.gz are:

• MD5: e0367c184a6d5b3085e3258ca0520610

• SHA1: a77edf79576039374a3d88872b9059ac5f6d25aa

• SHA512: a8efaa13 4d73554c fa3b5b59 94fdab6d 2ca7d943 d31058f8 318c6232 ea6f39fa 8c6496b9 bf0e318b 9a673f41 a2b58b2b 3ea36b46 3e98ac8c f3b125f0 3caa9e9c

Go to the downloads page to get it!

Posted by jmmv on Sun, 18 Jan 2009 20:20:00 GMT | Permanent link

After a very long quiet period, I am happy to announce the 0.6 release of ATF. The major goal of this release is to integrate the changes made to the source code by Lukasz Strzygowski as part of his Google Summer of Code 2008 project. Integrating these changes into a formal release is a prerequirement for later integration of his NetBSD-specific work on regression tests into the NetBSD source tree.

The major changes in this release are:

• Make atf-exec be able to kill its child process after a certain period of time; this is controlled through the new -t option.

• Change atf-sh to use atf-exec's -t option to control the test case's timeouts, instead of doing it internally. Same behavior as before, but noticeably faster.

• atf-exec's -g option and atf-killpg are gone due to the previous change.

• Added the atf-check(1) tool, a program that executes a given command and checks its exit code against a known value and allows the management of stdout and stderr in multiple ways. This replaces the previous atf_check function in the atf-sh library and exposes this functionality to both atf-c and atf-c++.

• Added the ATF_REQUIRE family of macros to the C interface. These help in checking for fatal test conditions. The old ATF_CHECK macros now perform non-fatal checks only. I.e. by using ATF_CHECK, the test case can now continue its execution and the failures will not be reported until the end of the whole run.

• Extended the amount of ATF_CHECK_* C macros with new ones to provide more features to the developer. These also have their corresponding counterparts in the ATF_REQUIRE_* family. The new macros (listing the suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and _STREQ_MSG.

The checksums for atf-0.6.tar.gz are:

• MD5: 55cf06177889dbe1891c766dc5bf5b94

• SHA1: 311b277477bae7dab8e634f7c17cce25a8664012

• SHA512: 4aa980c5 545e606d 1b660f05 a8eae399 12830606 9703ed09 5f6d30a3 a963af77 ea53a8a7 daada27c 8e77f13b a385ce28 70472791 ef2bf81f f9385fda 800525db

Go to the downloads page to get it!

Posted by jmmv on Thu, 01 May 2008 17:00:00 GMT | Permanent link

I am very pleased to announce the availability of ATF's fifth public release, 0.5, which is probably the one with the most code changes so far. The main goal of this release is to provide a C-only binding to write test cases, completely freeing the developers of the use of C++. This new C library weights at around 7000 lines and the C++ library has lost around 1000 due to the fact that it now reuses code in the C library. Another important change of this release is the relicensing of the complete code base under a 2-clause BSD license.

The major changes in this release are:

• Clauses 3 and 4 of the BSD license used by the project were dropped. All the code is now under a 2-clause BSD license compatible with the GNU General Public License (GPL). Read the new license for further details.

• Added a C-only binding so that binary test programs do not need to be tied to C++ at all. This binding is now known as the atf-c library.

• Renamed the C++ binding to atf-c++ for consistency with the new atf-c.

• Renamed the POSIX shell binding to atf-sh for consistency with the new atf-c and atf-c++.

• Added a -w flag to test programs through which it is possible to specify the work directory to be used. This was possible in prior releases by defining the workdir configuration variable (-v workdir=...), but was a conceptually incorrect mechanism.

• Test programs now preserve the execution order of test cases when they are given in the command line. Even those mentioned more than once are executed multiple times to comply with the user's requests.

The checksums for atf-0.5.tar.gz are:

• MD5: b1d078b716e344bcf65f735cd74dc5d8

• SHA1: 63cb8541608518cdb214a10e71228d3e8c8af514

• SHA512: f78cce89 bfa34810 9e4bbd5c 20ea1930 8f8e3c4a 835b22cc 60e0e300 d243f98a ab4180ac 28673767 f34bbddf 1ad36acc 7569f483 0773d391 c5a38df4 1d6883a2

Go to the downloads page to get it!

Posted by jmmv on Mon, 18 Feb 2008 11:15:00 GMT | Permanent link

The branch org.NetBSD.atf.src.c has just hit the public repository. This contains a preliminary implementation of a C-only library for test programs. It will allow developers using ATF in their projects to restrict their test programs to the C language without having to use C++ at all, which is a highly desired feature in, for example, the NetBSD source tree.

The main focus of 0.5 will be the stabilization of this branch and its integration into mainline, assuming there is no important bug fix to roll off before this happens.

See some more details in this atf-devel post. Feel free to raise your comments there.

Posted by jmmv on Thu, 04 Feb 2008 21:00:00 GMT | Permanent link

I am very pleased to announce the availability of ATF's fourth public release, 0.4. The initial main goal of this release was to provide some developer-specific documentation, but once I attacked the implementation of the features planned in the roadmap, things got more complex. In the end, this release brings several new features and lots of portability fixes.

The major changes in this release are:

• Added two new manual pages, atf-c++-api and atf-sh-api, describing the C++ and POSIX shell interfaces used to write test programs.

• Added a pkg-config file, useful to get the flags to build against the C++ library or to easily detect the presence of ATF.

• Added a way for test cases to require a specific architecture and/or machine type through the new require.arch and require.machine meta-data properties, respectively.

• Added the timeout property to test cases, useful to set an upper-bound limit for the test's run time and thus prevent global test program stalls due to the test case's misbehavior.

• Added the atf-exec(1) internal utility, used to execute a command after changing the process group it belongs to.

• Added the atf-killpg(1) internal utility, used to kill process groups.

• Multiple portability fixes. Of special interest, full support for SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 C++ compiler.

• Fixed a serious bug that prevented atf-run(1) from working at all under Fedora 8 x86_64. Due to the nature of the bug, other platforms were likely affected too.

The checksums for atf-0.4.tar.gz are:

• MD5: df5b8794577cf962ccb7a0434644c9a3

• SHA1: dce4885a2f2d55d73d5f248fe482d5ac3101ce86

• SHA512: 93bb92bb 50784092 5ba10613 240360e6 eef605c9 e1eafff7 c0bcdc4b b27c27cd b272d479 cedf3bee b37d7550 bcb58ab5 c1fdb893 04492277 9bec29c8 4c9659ce

Go to the downloads page to get it!

Posted by jmmv on Thu, 12 Nov 2007 21:20:00 GMT | Permanent link

As I mentioned yesterday, ATF was hopefully ready to be integrated into NetBSD with the release of the 0.3 version. And it was! As there seem to be no serious problems with it, and as some people were adding pressure to get the thing integrated, I merged ATF into the NetBSD-current source tree today. Or, in other words, ATF will be part of NetBSD 5.0!

This marks a big milestone in the project, because ATF is now really put into use by an important and huge project: The NetBSD Operating System. This will hopefully boost interest in ATF and raise external contributions to either the core framework and/or to the NetBSD-specific tests.

For more details please refer to my official announcement at the tech-userlevel mailing list or to the news item at the NetBSD site.

Posted by jmmv on Thu, 11 Nov 2007 21:45:00 GMT | Permanent link

I am very pleased to announce the availability of ATF's third public release, 0.3. This release marks a big milestone for the ATF project: it is planned to be the first one to be integrated into the NetBSD source tree. Integration into the NetBSD operating system was the original goal of the ATF project as part of a Google Summer of Code 2007 project but, as it was not good enough when the program ended (or at least I considered it was not ready), integration was delayed. It is now polished enough to take make this important step forward!

The major changes in this release are:

• Added XML output support to atf-report. This is accompanied by a DTD for the format's structure and sample XSLT/CSS files to post-process this output and convert it to a plain HTML report.

• Changed atf-run to add system information to the report it generates. This is currently used by atf-report's XML output only, and is later printed in the HTML reports in a nice and useful summary table. The user and system administrator are allowed to tune this feature by means of hooks.

• Removed the test cases' isolated property. This was intended to avoid touching the file system at all when running the related test case, but this has not been true for a long while: some control files are unconditionally required for several purposes, and we cannot easily get rid of them. This way we remove several critical and delicate pieces of code.

• Improved atf-report's CSV output format to include information about test programs too.

• Fixed the tests that used atf-compile to not require this tool as a helper. Avoids systems without build-time utilities to skip many tests that could otherwise be run. (E.g. NetBSD without the comp.tgz set installed.)

• Many general cleanups: Fixed many pieces of code marked as ugly and/or incomplete.

The checksums for atf-0.3.tar.gz are:

• MD5: b01dde98a3950f4cbdb914197455dab7

• SHA1: 3734ed79d6116e4ec3c90ca2cf5aca5bac86e5a1

• SHA512: ae353c4a b1356bea 92dbbf9f 53a8c1fb e381fffb e4f09959 3580926d 242be393 25043592 440a64d1 b14be895 ebef8ded 05716c89 04cc309c a6031933 8fc3c7ba

Go to the downloads page to get it!

Posted by jmmv on Thu, 20 Sep 2007 23:15:00 GMT | Permanent link

Exactly a month ago, the Google Summer of Code 2007 program concluded and, coinciding with that date, I published ATF 0.1. Development has continued since then and I am now pleased to announce the availability of the ATF 0.2 release, the first release outside of the program; hopefully many more will come. This new version has still many rough edges but given that the goals planned for it have been accomplished, I felt it was time to make it public.

The major changes in this release are:

• Test cases now get a known umask on entry.

• atf-run now detects many unexpected failures caused by test programs and reports them as bogus tests. atf-report is able to handle these new errors and nicely reports them to the user.

• All the data formats read and written by the tools have been documented and cleaned up. These include those grammars that define how the different components communicate with each other as well as the format of files written by the developers and users: the Atffiles and the configuration files.

• Added the atf-version tool, a utility that displays information about the currently installed version of ATF.

• Test cases can now define an optional cleanup routine to undo their actions regardless of their exit status.

• atf-report now summarizes the list of failed (bogus) test programs when using the ticker output format.

• Test programs now capture some termination signals and clean up any temporary files before exiting the program.

• Multiple bug fixes and improvements all around.

For a list of supported platforms in this release please see the downloads page. From now on this information will be kept there.

The checksums for atf-0.2.tar.gz are:

• MD5: 163331daef2a41caafda9f08dc7bf5cd

• SHA1: f31cf24df78c194b63c0694c0f1a7b2f7bb0c9b6

• SHA512: d6edbba3 15da21fc 9b8c6318 18c42aa1 0dc44d00 2af045bf 4bfd1e21 835912b1 5914b383 c84d32d6 48c4e421 10ed5049 460a3627 9016f14e e84e2905 37adb98a

Go to the downloads page to get it!

Posted by jmmv on Mon, 20 Aug 2007 21:40:34 GMT | Permanent link

I have just updated the first preview of NetBSD-current release builds with ATF merged in to match the ATF 0.1 release published today. As already stated in the old news item: These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF is imported into it; we are not sure when that will happen, though.

By the way, ATF 0.1 is also in pkgsrc as devel/atf. This may be an easier way for you to try it, even though you will not get the NetBSD-specific tests.

For more details see my post to the NetBSD's current-users mailing list.

Posted by jmmv on Mon, 20 Aug 2007 13:01:00 GMT | Permanent link

I am very pleased to announce the first public release of ATF, numbered 0.1 as you may expect. This release is being published to coincide with the end of the Google Summer of Code 2007, a program that sponsored this initial work in the project. Development will continue from now on with the primary goal of integrating ATF into the NetBSD operating system. But, of course, feel free to adopt ATF in your own applications.

This release has been tested under NetBSD-current (4.99.29), NetBSD 4.0_BETA2, Mac OS X 10.4.10 and openSUSE 10.2.

The checksums for atf-0.1.tar.gz are:

• MD5: e043545044fa3aa9da0dd2a62712e1b6
• SHA1: 3f3b8355d30f1372a5e3e5fed8621441956b6a4d
• SHA512: 5ae09aab a448cfe7 d01b807c e3159164 d6138c2c bc335b7b da537403 e6455e87 8ba737fe 72c2bbea a97d9f9f 75c9a0d1 d65ddf16 a3a66a47 cd84fba3 487e756d

Go to the downloads page to get it!

Posted by jmmv on Sat, 18 Aug 2007 16:32:12 GMT | Permanent link

In preparation for the first public release of ATF, which will be released on the 20th, the project has got its first mailing list addressed at covering development-related topics and, at the moment, user-specific questions too. Its name is atf-devel, and you can find more details about it in the mailing lists page.

The description for the list reads:

atf-devel is the mailing list to discuss development issues related to the Automated Testing Framework (ATF) and is not tied to any specific project using ATF. Given that ATF is still very young and that the list is low-traffic, this is also the appropriate place to ask user-related questions. Inquiries related to the integration of ATF into NetBSD should be directed to tech-kern or tech-userlevel instead.

Many thanks to S.P.Zeidler from the NetBSD administration team for setting this up so quickly.

Posted by jmmv on Wed, 8 Aug 2007 01:52:12 GMT | Permanent link

I have just uploaded some NetBSD-current release builds with ATF merged in. These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF 0.1 is made public, which should happen later this month.

For more details see my post to the NetBSD's current-users mailing list.

Posted by jmmv on Sun, 16 Jul 2007 08:00:00 GMT | Permanent link

The official web site for ATF was launched. The old page will live until the Google Summer of Code 2007 program is over. If you ever visited the web site of the now-dead Buildtool project, this page will look very familiar. I did not want to rewrite everything from scratch, so I just copied several files from there, modified them a bit and liked the results.

Oh, and by the way, ATF has also got a logo! See it in the top-left corner of the page. It is pretty simple and may be replaced if somebody else comes up with a better one. But it is cool-enough for now :-)

Posted by jmmv on Tue, 10 Jul 2007 17:00:00 GMT | Permanent link

ATF's code has been made public just in time for the Summer of Code 2007 midterm evaluation. You can find more information on how to access these sources (in early stages of development) at the repository page.

Posted by jmmv on Thu, 12 Apr 2007 10:00:00 GMT | Permanent link

NetBSD was in the need of a testing framework to replace its current home-grown tests. ATF was presented as a project proposal to resolve these needs, and Julio M. Merino Vidal (the one writing these lines :-) was selected to work on it. Stay tuned for further news.