The NetBSD Foundation 2004 Annual Report ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The NetBSD Foundation held its Annual Group Meeting of all members of the NetBSD Foundation, i.e. all NetBSD developers. From the more than 200 active developers, 65 made it to the meeting, which took place from 22:00 UTC to midnight on January 15th 2005. This is a report from the presentations of the various groups within the NetBSD project about their past achievements and future Goals. This report is mostly a verbatim version of the text presented in the online meeting, with a few places edited slightly, marked by "[...]". * Agenda 1. Welcome and Intro remarks (Christos Zoulas) 2. Presentations from the Executive Committees a. Technical exec (Alistair Crooks) - core (Allen Briggs) - security-officer (Daniel Carosone) - releng (James Chacon) - pkgsrc (Johnny Lam) b. Communications exec (Tracy Di Marco White) - www (Jan Schaumann) c. Finance exec (Lex Wennmacher) d. Membership exec (Thomas Klausner) e. Administration exec (Tracy Di Marco White) 3. Any Other Business 4. Closing remarks (Christos Zoulas) * Introduction by the President of The NetBSD Foundation Christos Zoulas, president of The NetBSD Foundation, first introduced the members of the current Board of Directors: - Chris Demetriou and Tracy Di Marco White are members of the board, - Lex Wennmacher is a member of the board and Treasurer, Luke Mewburn is a member of the board, Vice President and Secretary, - Christos Zoulas is member of the board and President of the NetBSD Foundation. Christos thanked both personally and on behalf of the Board of Directors the two departing members Chris Demetriou and Luke Mewburn for their hard work during the past two years. Chris led the Foundation as President two years ago (followed by Christos in the past year) and Luke is the current secretary and during the past year was instrumental in organizing and overseeing the selection of our new Logo. Next, Christos welcomed the two new Board members Alistair Crooks and Perry Metzger who will be replacing Chris and Luke and their term will begin at the end of this meeting. Finally he thanked the chair of the Nomination Committee Jim Wise and its members, as well as Thomas Klausner who was our Election Administrator and Simon Gerraty who was the Election Validator for the Board Election. Last, Christos thanked everyone involved in the Executive Committees who spend a lot of time, both visibly and behind the scenes to keep the Foundation running. His congratulations went to the pkgsrc members for constantly improving a very high quality product, to the www team for keeping our website up-to-date, to the systems administration group for persevering despite all the hardware failures, and to the release engineering group for managing to push 2.0 out of the door. His personal thanks also went to Lex and Tracy for taking over a lot of what used to be his responsibilities and doing a better job than him. Christos wished the new Board members Alistair and Perry a successful and fruitful Board term. After he was done with the introductions Christos gave a bit of a personal perspective to his involvement with NetBSD that has been going on for the past 12 years. Christos Zoulas: ``I joined NetBSD after lurking in the in the mailing lists because I was truly impressed by the level of technical expertise and the quality of the source code. I started as a developer who did not know left from right on how anything in the kernel worked (and some might say that I still don't) and ended up being the President of the Foundation. I am saying all this because I feel that our future depends on our ability to attract, recruit, teach, organize, and guide new talented people who are going to be the future of the project. As a volunteer effort we depend on the generous time and resource offers of the members our community. We all have other obligations in our lives - both personal and professional - which take precedence to NetBSD, so I understand how difficult it is to allocate time on a consistent basis for a volunteer program. I can say that my involvement has been very rewarding for me both in improving my technical knowledge, and making me feel appreciated for contributing to something that I do for fun and not for financial gains. I want to encourage everyone to think of ways to become more involved with the running of the project. I value your opinion and need your help to improve NetBSD. It is easy to sit back and complain about our faults and deficiencies, but a lot harder to fix them. As a community we excel in criticizing everything but when it comes to take responsibility to making changes and fixing the problems only a few of us step up to the plate. Working on an Operating System is very challenging - more so now than before. Ten years ago neither Windows or Linux were serious competitors both in the functionality and stability axes. Now both offer more features than we do, and they have behind them the resources of very large commercial organizations. If that was not enough competition, there is a plethora of other operating system choices each trying to fill a niche. Even our close siblings FreeBSD and OpenBSD have developed certain features we still lack. Are we becoming irrelevant? Is it time to give up? I don't believe that either is true. I see our competition facing challenges of their own. Linux keeps re-writing major portions of the kernel and has stability issues. It now depends on 3rd party vendors to integrate and make stable releases of the code. FreeBSD took over the huge task to implement fine grain SMP and after two years of effort they still don't have a production quality system. OpenBSD is still touting its security features but lacks the manpower to integrate major kernel features such as UBC and address performance problems. Instead it focuses in supporting and re-implementing major userland utilities. The Windows release cycles keep getting longer and longer and promised features keep getting postponed because of the increasing complexity of the operating system. Sun is trying to keep Solaris relevant by open-sourcing it, but nobody is certain of what is going to be open-sourced and when. Apple's Darwin effort does not seem to be producing any useful results, possibly because it is not complete, and the open-source version of the tree is always behind the commercial version. So where does that leave us? What is the purpose of NetBSD? I see a lot of opportunities ahead for us. We've managed to release 2.0 this year and we have integrated major features without serious regressions or performance issues. We can leverage a lot of the work in FreeBSD and OpenBSD to improve some of the areas we are lacking. We have the cleanest source tree and our wealth of hardware support makes us a very attractive porting target. Our cross-building capabilities have made it possible to support the plethora of ports with much less effort than before. Our BSD style license allows vendors to use our code without suffering the intellectual properly loss dilemma presented by the GPL. We have our work cut out for us and we need work hard in the areas we are behind in order to remain relevant. What are our challenges ahead? Our goals are divided broadly in three categories. The first category is technical: - We need a better filesystem. We need a filesystem that will support the ever growing disk sizes, does not need hours of fsck after a crash, supports snapshots, replication and comes with a volume management system that allows us to add and remove devices dynamically. It would be nice for the filesystem to be able to handle hardware faults gracefully by keeping multiple copies of critical data, and being partially accessible. - We need real-time scheduling support, POSIX real-time extensions, and thread-safe libraries. There is an increasing number of applications related to video, voice, and control that need hard real-time support. We can start by bullet-proofing our thread support and finishing the re-entrancy issues with our libraries, then continue by evaluating real-time scheduling and making subsystems of our kernel able to use multiple processors. - We should modernize our network stack. Features such as SACK and ECN are becoming necessary. We should support multiple default routes, load balancing and throughput control. We need to fix our firewall software immediately and in the longer run come up with something cleaner and more extensible. - We should improve our internationalization support. - We need to replace our Loadable Kernel Module support with a real kernel linker that will allow us to make the kernel more dynamic. - We need better ACPI support that includes suspend and resume. - As a longer term goal we should look into clustering, process migration, and redundancy support. - We should take advantage of pkgsrc to provide workstation or server type systems with groups of software that can be easily installed on top of the base installation semi-automatically. I don't think that I am saying anything new here. The second category is administrative: - We need to produce more frequent releases. We must provide timely response to security issues and have a better mechanism to supply patches to supported releases. - We need to institute better testing and try to provide more timely feedback and concentrate on closing important bug reports. We should also improve our installation process. While it is easy to install NetBSD in our more popular platforms, it can be pretty daunting to install on the more exotic ones. - We need to make our committees more responsive and transparent. We need to institute and document policies, protect our intellectual property and trademarks, and get more people involved. We have too few people in charge of many different positions and we constantly fail to respond to the needs of the project. - We need to improve our system services and administration support. The third category is publicity, fund-raising, and the project's image: - We should setup fund-raising goals, and communicate what we are going to be using the funds for. - We should organize developer get-togethers and presence in major conferences. - We should contact the commercial users of our software and encourage them to publicize their use of NetBSD as well as feedback fixes and improvements to the code base. - We should encourage more developers to join our project as well as help them during their first steps, without intimidating and criticize their work needlessly. But we need your help to achieve all of that. And this help is not limited in the technical areas. We need more people in the project management and communications areas where traditionally we have shown less focus. Let's make this year different. Let's make the 3.0 release the best one yet and on time. Let's make 2005 the year where YOU are going to get more involved and make YOUR forward in the right direction. Thank you.'' After Christos Zoulas' general overview, each presenter talked about the things their group achieved in the past year. * Introduction of the Technical Executive Committee The overview of the Technical Executive Committee was given by Alistair Crooks: ``The Technical Executive Committee is in charge of four areas: the core team, release engineering, security-officer, and pkgsrc. Its current members are: Alistair Crooks, Lex Wennmacher, and Luke Mewburn. [...] Over the last year, there have been a number of achievements made by the various technical teams which come into the technical executive committee's area. I will let the various PMCs talk a bit about what has been achieved, any pressing issues or concerns which they have, and a bit about how they see the future over the next year. They have all worked very hard throughout the year in their own individual areas of expertise, and I'd like to commend and thank each team for their work.'' * Report of the Core Team The core team presentation was given by Allen Briggs: ``Current core members are the same as last year: briggs, christos, fvdl, lukem, matt Random note: * Core works on a rotation basis and attempts to get a majority opinion for official rulings. This works pretty well most of the time, but can be slow to collect responses from everyone. * [...] Some technical highlights, many since the 2.0 branch: * statvfs(2) * pf * Major ipf updates * New ports * Many software updates (postfix, bind, groff, etc.) * Many manpage improvements * Many thread improvements * Many audio improvements * Many port improvements * Version numbering changed with 2.0 release * ptyfs * PAM coming in now 2.0 goals that we missed: * GNATS scrub. We still have a lot of outstanding PRs. Some folks have done a great job in cleaning up some old ones, but we really need to find a good way to do better as a project. Suggestions welcome. * Libraries all thread-safe. There has been a pretty good improvement here, but there's almost certainly more work to be done as was so tactfully pointed out recently on the mailing lists. Core concerns: * Outstanding PR list. More than 12% of the PRs in our database are still 'open.' We'd like to see that significantly down by next year. * Ports with inactive port-masters. There are some ports that have been more or less abandoned. These should probably be picked up or mothballed. If you love a port, please nurture it. * Better organization of project roadmap. We're just getting started with a roadmap. It would be nice to extend and subdivide it to cover individual ports as well as different kernel and userland subsystems. This might be a way to help re-engage some port-masters. * Testing. It would be nice if we can develop a lab or set of volunteers to do some automated or at least regular testing for functionality and performance. * Long release cycles. The 2.0 release cycle was too long. We'd like to find ways to work with releng to help reduce the release cycle time in the future. In addition to trying to address these concerns, we're looking toward the future. The roadmap shows a number of things that we'd like to see by 3.0, currently scheduled to branch at the end of February -- if you've offered to do these things, please try to get them done or 3.0 will happen without them (I include myself in that admonishment): * Implementing get*ent_r() and getXbyY_r() * Flesh out wedges some more and start taking advantage of them * Better ieee1394 support * Updated GDB * Modular nsswitch * Complete PAM integration * Better internationalization * TCP/SACK * EtherIP * Merge of the ktrace-lwp branch * POSIX shared semaphores * XFree86 4.5 Looking further ahead, our roadmap includes: * NFS4 * System packages * Modular scheduler * Updated GCC (4.x branch, if it's stable) * Generic device hot-plug * Better kernel locking primitives ("newlock") * Migration to X.org from XFree86 * TCP/ECN * Better support for modern parallel ports * AIO support Of course, these items and those suggested by Christos earlier are only possible with the good work of all of the developers. Thank you for your efforts throughout the years. It's a pleasure working with you all, and I look forward to seeing what NetBSD can become!'' * Report of the Security Officer The security officer presentation was given by Dan Carosone: ``In previous years, Security Officer has provided details of the following matters in our presentations. For more detail on these, see the AGM logs [...]; today we'll provide a quick update, for those entries which are likely to change over time. * What does the Security Officer team do? * Role of Security Officer team * Responsiveness * Projects * Who is Security Officer? (dan, david, groo, itojun & perry) The security-officer role is intended to be primarily a coordination and management function, providing a public contact point for NetBSD security matters, and marshaling the project's resources, members and activities towards various security-related goals. There are two major aspects to the workload: incident-driven and development-driven. This past year has, happily, been a relatively quiet year for NetBSD security incidents (less so for issues that affect pkgsrc). Unfortunately it's also been somewhat quiet for some of our development goals as well. We laid out a number of goals at last year's meeting. We've made mixed progress on these during the year. For each of these, we'd like to give a self-appraisal and summary of progress last year. Last year's goals ----------------- Goal: Continue to improve Binary Patches for Advisories Appraisal: A Summary: Binary patches issued for most Advisories (where applicable). * More automation to be done yet. * Thanks to Releng for access to autobuilds Goal: Re-organize Advisory publishing process * Make CVS handling cleaner * Lessen publication workload on S-O Appraisal: B Summary: CVS handling now seems to work well. Automation of mail-out process helps greatly. * Some 'toolchain' and familiarisation issues with updating htdocs (more details below) Goal: Get >75% of active developers to have PGP keys * Helps for sharing confidential information with relevant expert developers * Increases security awareness and tool familiarity among developers * Increases visibility of NetBSD among security savvy non-developers Appraisal: B+ Summary: 104 developers' PGP keys (in localsrc/security/publickeys/developers) * Good degree of connectivity in web of trust * All new developers will have PGP keys * Has been useful in increasing the trust for admin requests (eg, resetting ssh access) * Not 75% yet, but getting closer; conferences and drinking events seem to help and more are encouraged :) Goal: Security-Officer web page updates * Help people understand who S-O is, does Appraisal: F Summary: Web Pages not significantly updated * See 'staffing' below Goal: Improve mail response time, Advisory publishing time Appraisal: A- Summary: Response to first contact email mostly quite fast * Dropped the ball on very few issues - ( 1? ) * See 'staffing' below * Fewer Advisories this year, fortunately Goal: Improve internal S-O tracking and communication Appraisal: B Summary: Internal communication has become very good No tracking system yet; delayed by a number of infrastructure issues; working with admins now and hope to start testing a new one shortly Goal: Split Security-Announce from NetBSD-Announce Appraisal: F Summary: Security-Announce list created, but not cut-over * Remains outstanding * Questions about process and user preferences * Was a lower priority, given available staff time and workload Goal: Better communication with Releng Appraisal: B+ Summary: Regular exchanges between Releng and S-O via mail and chat provided better coordinated response. The one thing guaranteed to bring up security issues is an imminent release. Goal: Sign and Publish ssh host keys and PGP keys Appraisal: B- Summary: key tables were created, and made available via ftp * Web pages not updated yet * Coordinate with admins to keep host keys updated Goal: Sign Releases Appraisal: A Summary: 2.0 release included a signed set of file hashes * SHA1 and MD5, signed by s-o key * Need to automate creation and maintenance Goal: Establish and publish "Security Notes" (quick 'NetBSD is not affected' announcements) Appraisal: A- Summary: The Security Note format was used, and well received. * Need to update web pages to index them Goal: Extended team of security volunteers, to help handle non-confidential issues, and contribute time to handle less critical flaws which are public and not yet SA'd and to follow up with CERT and provide NetBSD references for older CERT issues Appraisal: F Summary: We received only one inquiry from a volunteer this year offering to help with ongoing S-O work * We did solicit help on specific issues from specific experts and groups like port-masters * We didn't make a general plea to developers This Year's goals ----------------- * New Tracking tools * Update web pages * Improve updates to host keys files, with admins help * Improve, extend Security Notes * Extended team of security volunteers Web Site updates ---------------- The www team has been completing the process of converting the NetBSD website to XML. Making updates now needs a number of new tools installed, and a number of different steps - which have changed as the site develops. Overall, this is a hugely positive change, but at least some of us have had 'awkward surprises' when trying to update the website to publish a new Security Advisory. Regardless of the internal format, there are considerable updates required to the Security/ web page static contents, updating and better documenting NetBSD's security presentation to the public. Although we hope to be more successful with that work this year, please feel free to make your own contributions. For some time, we've also had a desire to improve the organisation and presentation of the more data-driven aspects, such as automated indexes linking SA's to affected releases (and, later, syspkgs). There are a number of aspects to this we haven't had time to work on. In the meantime, the VuXML project has developed a number of useful formats and tools we want to look at carefully. Staffing -------- Like other developers, all the security-officer team members have non-NetBSD work duties - 'day jobs'. This year, more than previously, availability of time has affected S-O. For most of the year, although quick incidents and issues were dealt with easily, few were able to contribute significant time to 'heavy lifting' incidents or projects. Therefore it was fortunate that there were fewer issues that affected NetBSD itself this past year. Many of the issues notified to security-officer were either not relevant to NetBSD, or were problems in third-party code that affected pkgsrc. The pkgsrc team did a great job in responding to those, and publishing the issues in the audit-packages mechanism; they have our sincere thanks. Security-officer always relies heavily on other NetBSD developers for the actual resolution of many issues. Several times this year, developers have found an issue in the course of other work, and notified security-officer complete with a draft Security Advisory. This is wonderful, it makes our job much easier, and greatly improves the speed at which we can publish. Thank you, again, and please keep it up! Even so, given other commitments, we can always use more help. There are several ways you can help listed below. What can you do to help S-O? ---------------------------- * If you have the time, contact us about helping with the extended team. It will have more relaxed urgency, and will help free up S-Os time (and prevent burn out) for handling critical issues. * Suggest and perform security sweeps for your pet issues. * Review PRs in security category - and close them! * Generate example systrace policies for system daemons. * Write rc tweaks to run more daemons unpriv'd/chroot/etc. * Stomp out more suid programs. * Volunteer to join S-O! Some of us are, or will eventually be, looking to retire and make room for new members.'' * Report of the Release Engineering Team The release engineering presentation was given by James Chacon: ``Currently, the releng team consists of the following developers (in alphabetical order of username): - agc Alistair G. Crooks - cjs Curt Sampson - cyber Erik Berls - grant Grant Beattie - he Havard Eidnes - itojun Jun-ichiro itojun Hagino - jdc Julian Coleman - jmc James Chacon - jwise Jim Wise - lukem Luke Mewburn - msaitoh SAITOH Masanobu - tron Matthias Scheler Before I get started too much into things I'd like to make sure and give a public thank you to everyone who worked on the 2.0 release. As I'll detail below this was a very long process for us and the team deserves many thanks for getting it done. Now onto the specifics. Major achievements last year: - We got 1.6.2 out the door in March. While this was mostly a maintenance release it did also address a number of security advisories as well as numerous kernel fixes. - We finally (after almost 8.5 months) released 2.0 to the public. I'll go into more detail below as to what we hope to do in the future to prevent these types of delays. - We'd like to thank everyone involved in the src/x11 cross-over X11 build support. I won't try and name people specifically here because I know I'll miss folks. The main thing is to highlight just how much easier this makes the release engineering process as everything that we put into a release directory on ftp.NetBSD.org is now able to be done from one source tree. - Maintenance continued on both the 1.5 and 1.6 source trees as pullups were submitted and processed. (though I will note the number of 1.5 submissions has been trickling off for some time). - Increased communication with the security-officer team. This has been problematic in the past as often times security issues pick the worst timing to show up (as we're releasing is often a good time it seems..), so keeping our schedules more coordinated is helping both teams out here. - Better communication with the rest of developers via requested semi regular status reports during a release cycle. Plans for this year: - Start the release of a security/critical fix branch from each main release. As detailed recently to netbsd-developers this sub-branch will consist of security/critical fixes only and is expected to get us binary release updates that security officer also needs when publishing an SA. It's expected that 2.0.1 will release soon (as the new autobuild hardware is installed and sysinst support for updates is added into the tree). - Release 2.1 sometime during spring of 2005 - The return of regular snapshot builds once the new autobuild hardware is in place. - Branch 3.0 at the end of February. NOTE: this doesn't mean we plan on overlapping 2.1 and 3.0. It's expected that 3.0 won't be ready until probably June so at least a month or more will exist between these releases. - Release 1.6.3 during 2005. It's unclear at this time when a good point in time for that will be, but as it may be the final release of the 1.6 tree it's release most likely will happen shortly after 3.0 releases and the 1.6 branch is closed down. - De-support/close down the 1.5 branch. There are no pullups currently waiting on this branch and an announcement about it's status should be going out shortly. Major hiccups/problems during 2004: - Obviously to most of us 2.0 took way too long to get into a releasable form. There were a large number of factors that led to this but the main thing that hamstrung us early on was getting ipf into a stable state that was ready for release. While it was critical to get onto the ipf 4 code in the future the initial branch should be pushed out if a major import like this happens at the last minute. - As everyone is aware this is a volunteer project so people have "real life" to worry about too. What this can mean when we drag a release out is conflicts arise with real life vs project schedules. So doing anything we can to keep things moving and on schedule helps everyone avoid stress :) - In addition we lost both releng machines during 2004. The main machine we published snapshots and did request tracking on (releng) died and during the build cycle for 2.0 it was found that the autobuild machine (tgm) was having memory timing issues. As a result of this new hardware was purchased and will be installed shortly. Things to improve on we need help with: - Portmaster involvement. We need the portmasters to be more involved with the functioning of their ports. This includes testing out the branch in enough time before the release to ensure there are no MD issues - Testing. More testing and feedback is desirable. Especially the case with pullups during the release cycles. - Bug fixing. As part of the 2.0 release we tracked a given set of PR's highlighted as "critical" to fix before release. Yet, some of those would sit for months before getting worked. This puts release engineering in a bind as we don't want to release code with known critical bugs, but without people stepping up to work bugs it makes things drag out for a long long time. - More manpower never hurts. Especially as we try and ramp up the critical/security branches for release having additional people willing to commit the time to coordinate releases will be crucial.'' * The pkgsrc team The pkgsrc presentation was intended to be given by Johnny Lam, but, unfortunately, he couldn't make it. Thomas Klausner gave the presentation for him: ``The pkgsrc PMC is the Project Management Committee which oversees the development of pkgsrc, the NetBSD Packages Collection. The pkgsrc PMC currently consists of Alistair Crooks, Thomas Klausner, and Johnny Lam (agc, wiz, jlam). Progress made in 2004 ===================== pkgsrc continues to grow in the number of packages supported in the tree, and in the number of operating systems to which pkgsrc has been ported. Particularly noteworthy progress has been made in the following: * Pkgsrc documentation converted to XML and placed in pkgsrc/doc/guide. This replaces the older monolithic Packages.txt, and we can now generate pkgsrc documentation in different formats, e.g. HTML, PDF, etc. [grant, jschauma, hubertf] * Full switchover to buildlink3 and the removal of all buildlink2 code from pkgsrc. This makes our infrastructure cleaner and eases pkgsrc development. [wiz, snj, minskim, xtraeme, and many other developers] * Wrapper scripts broken out of buildlink3 into a standalone framework. This was an important step toward making it easier to port pkgsrc to other operating systems and to older releases of NetBSD. [jlam] * Ports of pkgsrc to additional operating systems: - Interix (Microsoft Services For UNIX) [tv] - DragonFlyBSD [Todd Willey from DragonFlyBSD Project] * bootstrap-pkgsrc was merged into pkgsrc, which simplifies the task of getting pkgsrc running from scratch on any of the operating systems supported by pkgsrc. * A pkgsrc regression suite was added into pkgsrc/regress to improve our quality control for commits to pkgsrc. [gavan] * A build-time options framework has been added to pkgsrc to more uniformly describe the optional components that may be built into packages. [jlam] * Better execution of branching process -- we stuck to our goals of have well-defined limits to pkgsrc "freeze" periods, and we have managed to branch regularly at the end of each quarter for the past 4 quarters. [pkgsrc-pmc] The following items are part of a good trend -- the involvement of more pkgsrc developers in helping to maintain pkgsrc instead of simply committing new packages. More developers were given the freedom to take on new responsibilities: * Improved management of the pkgsrc-stable branch with the establishment of a releng-pkgsrc team. This team uses the usual releng tools to manage pullups to the pkgsrc-stable branch in a timely manner. [admins, releng-pkgsrc] * Much more responsive pkg PR handling thanks to a rotating team of developers (modeled on www@) that routes incoming PRs to other developers most likely to be able to handle them. [pkg-bug-handlers] * Dedicated groups of developers using pkgsrc on non-NetBSD platforms handle platform-specific PRs. This helps ensure that pkgsrc support for these platforms continues to improve over time. [solaris-pkg-people, darwin-pkg-people, linux-pkg-people, irix-pkg-people, freebsd-pkg-people, interix-pkg-people] * Continuous bulk building of pkgsrc on several different platforms to expose broken packages or broken infrastructure in a timely manner. [krister, agc, minskim, jschauma, and many others]. Thanks go to wiz for providing diffs of the bulk build results. Work in Progress ================ * tv-recurse branch has changes to pkgsrc infrastructure make calls so that we aren't as reliant on recursive make invocations. This will allow for substantial speedups when building large numbers of packages. [tv] * Package views is still considered experimental pending a solution to the problem of shared directories across packages. Two different ways of solving this problem were discussed in November 2004 on the tech-pkg mailing list, and both solutions have developers (jmmv, jlam) committed to implementing them during the course of 2005. Other noteworthy items ====================== * Thanks go to jmmv and markd for their respective high-quality jobs in updating the GNOME-2.x and KDE-3.x packages in pkgsrc to the latest versions. These are huge collections of highly visible packages, and they have good maintainers. * Sun has donated hardware for pkgsrc development. * Sun will be using "some form of pkgsrc" for its contrib packages extras in Solaris 10. * pkgsrcCon 2004, hosted by wiz & dillo at TU Wien, was deemed a success. pkgsrc developers and users were able to get together and share ideas/concerns/projects over a three-day period. pkgsrcCon 2005 will be hosted by salo in Prague on May 6-8 to hopefully start a tradition of being a well-spring of pkgsrc projects for the coming year. * DragonFlyBSD has some strong advocates within the project for using pkgsrc as its package system instead of FreeBSD Ports or rolling their own. This is a tribute to the excellent job that pkgsrc developers are doing to produce a very good and very usable package system, and an acknowledgment of how well we support non-NetBSD systems. Plans for the future ==================== * Continue to work on using cross-compilers to help build pkgsrc for slower machines on much faster ones. reinoud has posted a solution using distcc to tech-pkg@, and kristerw presented interesting work at pkgsrcCon 2004 that will hopefully be integrated into pkgsrc in the future. * Decrease our bulk build times by using clusters to spread the load. jlam will be presenting his work on this at pkgsrcCon 2005. * Be more responsive about security problems, perhaps by creating a group similar to security-officer for pkgsrc. * Improve bulk build results on non-i386 architectures. The two items above will help this greatly. * Usual stuff: - Create more packages! - Maintain your packages! - Port to more platforms! - Attract more pkgsrc developers! Where you can help ================== * If you have knowledge of how a particular feature works in pkgsrc, or if you add a new feature to pkgsrc, please try to write a regression test to test the proper functioning of that feature. This will help keep pkgsrc working as expected by allowing developers to test that their commits don't fail the regression tests. * Consider joining any of the various teams that help out with handling PRs, writing user documentation, or managing the pkgsrc-stable branch. * pkgsrc-wip continues to grow and to attract new developers-in-training. It would be very helpful to the pkgsrc project if more developers could find time to look over requests for package reviews on the pkgsrc-wip mailing lists. This helps to spread the knowledge, and to gain more people capable of answering pkgsrc developer FAQs for new pkgsrc-wip developers and users.'' * The Communications Executive team The report of the Communications Executive committee was given by Tracy Di Marco White: ``The Executive Committee for Communications Members: Tracy Di Marco White , Hubert Feyrer , vice chair, Jan Schaumann , Luke Mewburn , chair, Perry E. Metzger , Jeremy C. Reed , Scott Reynolds Review of the past year: * A new logo was selected out of over 400 submissions by 238 artists. IP transfer was handled by Luke, and now TNF owns its new logo. * We've been handling various announcements, coordinating with release engineering for the 2.0 release announcement, the trademarking for NetBSD, etc. * Jan has done a marvelous job coordinating the quarterly reports, advertising what we're doing so that people are made aware of our developments. * Making sure various news sites such as slashdot, daemonnews, osnews are made aware of NetBSD related news by submitting news items to them. * Put together an org chart of the TNF structure. http://www.NetBSD.org/Foundation/structure.png * Hubert worked with the German vendor JF Lehmanns (www.lob.de) to have CDs available in Germany and Europe. * Coordinating, helping, and performing presence of the NetBSD project at various tradeshows, and advertising them on http://www.NetBSD.org/gallery/events.html * Hubert has acted as point of contact for the benchmark Gregory McGarry did, measuring performance of FreeBSD 5.3 and NetBSD 2.0 providing webspace (http://www.feyrer.de/NetBSD/gmcgarry/), and making the article public for Gregory. * Hubert followed up discussion on NetBSD benchmarks by posting a list of comparisons of NetBSD to other operating systems, and posting the list to tech-perform@, see http://mail-index.NetBSD.org/tech-perform/2005/01/12/0000.html * Jeremy has converted the logo into a format cafepress is happy with, and a NetBSD t-shirt is available. The price will be set to generate funds for TNF. http://www.cafepress.com/netbsd Issues we'd like improvement on: * Support for active marketing. Hubert wanted to print official NetBSD stickers for distribution, but the budget wasn't available. * Better participation and flow of information from developers: documenting things properly (src/doc/CHANGES, htdocs) so our users can find documentation about things; * The cafepress free store only allows one item of each type. We'd like to be able to have more. Goals: * Fundraising Fundraising Fundraising * Establish ourselves as a point of contact for announcing "interesting" things, taking data from developers (see "participation and flow of information from developers" above) and preparing information to hand over to the www@ team, the netbsd-news@ and/or netbsd-announce@ list as well as various news sites (slashdot, daemonnews, OSnews, ...) * Hubert has established a personal blog with NetBSD related news items that were not achieved by TNF but that are still interesting to NetBSD users/fans. Advertising these to the NetBSD community and beyond is good public relations. * Several people are starting to do various tradeshows. Discuss organization steps to help each other, and as comm-exec@ contact t-shirt and CDROM vendors to donate items for handing out at these events. * Calendar with trade shows and events for upcoming 12 months and find developers or volunteers to begin registration and planning for events well in advance. (For example, LinuxFest Northwest is coming up in April.) * Build media contacts list for mainstream media: Dr. Dobb's, InfoWorld, Network World, etc. (They have covered BSD news a few times each year.) * Media relations schedule: send out at least one press release each month to various mainstream tech media (not just small BSD-specific sites). * Work better with release engineering to know when a release is out. Mainstream press needs lead time to be able to announce our releases. * Work with developers to co-write technical articles for technical magazines and technical events. Build a list of articles that should be done, target (magazine or event), and developers to help. * Continue advertising NetBSD and support people who do so. * Fundraising Fundraising Fundraising Thank you, and please consider helping promote NetBSD and pkgsrc.'' * The WWW team The www team presentation was given by Jan Schaumann: ``As in previous years, let's start out with who the www-team currently is. While there have been the same regular members on and off again, at the moment the rotation consists of: * D'Arcy J.M. Cain * Klaus Heinz * Jan Schaumann * Jeremy C. Reed * Simas Mockevicious Daniel de Kok , who has been a very active and helpful member of the rotation, regrettably had to drop out recently due to other commitments. These members take turns in answering the various questions that are sent to www@ and perform the regular maintenance work. More formally, the duties of the www team are outlined at http://www.NetBSD.org/People/groups/#www and http://www.NetBSD.org/developers/www.html. To summarize, we're somewhere in between of ``customer-service'' / ``tech-support'' and HTML-monkeys. Aside from that, the www team also coordinates with communication-exec and the general developer body to prepare official announcements not only on our own website, but on other forums as well. The www team also responds to mail coming in on the mirrors@ address and maintains the list of our various volunteer mirrors in coordination with the admins team. So what have we been up to over the last 12 months? In preparation for last year's annual meeting, we had set out a number of goals, and while we have tried to work on these goals, we were not able to accomplish all of them: * translation work has stagnated in some areas / languages but also improved in others. The Korean and Lithuanian translations (maintained by minskim@ and symka@) are currently the most active translations efforts. * news announcements and general advocacy through our website has, I believe, improved somewhat, and cooperation with other committees is smoother now as well. * we did *not* manage to get any work done at all on projects.NetBSD.org. As was pointed out last year, this server contains a large number of completely dormant projects. These should be updated. * developers' public keys have *not* yet been made public on the website. This is an item that has completely dropped off the radar and we probably should file a PR for this to remind us. So, you may ask, have we done nothing useful at all in 2004? Well, we did achieve a few things over the last 12 months, and I feel that we are still continuously improving: It is worth noting that the NetBSD guide (basically a project all on its own) has been revamped and updated for the 2.0 release. The overwhelming majority of the work in this area has been done by Hubert Feyrer, who we owe much thanks for this. We have published the new logo together with communication-exec. While this of course lead to a lot of discussion, the process of updating the entire website and virtually every single page has shown us weaknesses in our infrastructure. At the moment, there are 2244 HTML files in the htdocs repository. Of these, 299 are generated from XML (178 at the time of the publication of the logo) and 396 from .list format (via perl). So for any global change, we still need to hand-edit 1549 files. Yuck! This is why we have focused in the last quarter of 2004 on the conversion of the existing pages to XML. While this brings its own difficulties, it has a number of convincing advantages: * all pages generated from XML have the same consistent look and feel * if the need arises, we can easily change the look and feel of the entire website * common segments (header, footer, styles etc.) can be controlled in one central file, so that a simple 'make' in the root can regenerate all files * all files are automatically standards compliant Some of the problems caused by the use of this framework are the large number of packages that each developer needs to have installed. Another problem is that different versions of the required packages produce different output, leading to a number of unnecessary 'regen' commits. This has lead to some ``awkward surprises'' and frustration among developers who have to touch htdocs occasionally. We are understanding of this frustration, recognize the shortcoming and have started to work on a remedy for this situation: We have started to change our point of view of the files under CVS to accept that all files that are generated from another source, should be treated as ``object files''. This means that they should not really remain under CVS control -- instead, developers should just commit the source files and be done with it. I'm currently working on setting up www.NetBSD.org such that it can build the htdocs tree itself. This setup is virtually complete and currently in a testing phase. Once we're satisfied that everything works as planned, we can remove the .html files from CVS. This has the advantage that it doesn't matter anymore which version of which required package a developer has installed. In fact, developers who routinely do not do htdocs work can just commit trivial changes to the source files without the need to generate the html files themselves. The setup on www.NetBSD.org also acts as an implicit error-checker: if a commit breaks the generation of the website, it will complain and the website will _not_ be updated. So, to summarize (and reiterate), while we understand the frustration with the difficulty of using the XML framework, we hope that the developer community understands that in the long run it will make things Much Easier. And while we did not achieve many of the individual goals set out last year, I nevertheless believe that we accomplished _some_thing and had a somewhat successful year. General goals for the coming year, goals the achievement of which each developer is encouraged to contribute to, are: * remove redundant documentation and eliminate duplication: There are many pages that contain similar information. They should be merged with other documents, moved into the guide or purged altogether. * continue to convert files to XML * revive / continue with translation work: Each developer who is even semi-fluent in any languages other than English is strongly encouraged to help with this task. * create a framework to manage ports-pages wrt to new: releases and news items: the 2.0 Release pointed out that at the moment there is no simple way of updating the largely identical information on ~60 pages. * entice port-masters to keep their port-pages up to date: Many of the port-pages have not been updated in *years*. Surely there has been some progress or some noteworthy achievement, yet the www team can not keep track of all the changes in all ports. * consider a redesign of the main page: The introduction of the new logo has lead to some discussion about the general design of the main page. While many users like the simplicity of the layout, it has become clear that none of us is a web designer. Several users (and developers) have made suggestions as to how to improve the websites layout, but the danger of (yet another) huge bikeshed discussion has inhibited any progress. Which actually is too bad. And with these suggestions for discussion and goals, I'd like to close the presentation of the www-team. I'd like to thank all the members on the www team for their hard work and good spirits.'' * Financial Executive Committee Lex Wennmacher presented the report from the finance committee: ``The duties of finance-exec are managing the financial business of the Foundation. The current members of finance-exec are Christos Zoulas (chair), Alistair Crooks (vice chair), Lex Wennmacher (treasurer), and Andrew Brown (assistant treasurer). Our annual expenditure is mainly on servers, disks, and miscellaneous hardware, but also on legal fees. Currently all of our expenditures have to be balanced by our only income, donations. Since the last Annual General Meeting, we received about 65 donations, some were quite generous. We acknowledge donations on our web pages, if the donor so wishes. The release announcement for 2.0 contained a solicitation for donations which was remarkably successful, as we noticed a significant donation increase. This concludes finance-exec's presentation.'' * Membership Executive Committee The presentation of membership executive committee was given by Thomas Klausner: ``Membership-exec oversees all aspects of membership in the Foundation. The current members of membership-exec are Lex Wennmacher (chair), Christos Zoulas (vice chair), Tracy Di Marco White, Thomas Klausner, and Ken Hornstein. The routine work of membership-exec consists of handling and deciding on new applications. Since the last Annual General Meeting, we have received 17 applications, had 12 new developers, and 2 resignations. We lost one developer, Eric Reid, in a tragic accident. One of the principal duties of membership-exec is keeping the records relating to the membership of the Foundation, used for example when preparing the list of Active Members for determining who may vote in Board Elections. Our bylaws require that developers must have signed a Membership Agreement before they can be considered members. Another achievement in 2004 was the finalization of an improved Membership Agreement - the old version of the agreement didn't properly reflect the new structure of the Project and was legally problematic in several ways. For this year we've got the following items on our agenda: - implementing a developer contact database (containing postal addresses and phone numbers) - developing a procedure for validating developers when they lose their ssh key - developing a procedure for resignations That's all, thanks for reading.'' * The admins team The presentation of the Executive Committee for Administration and the Project System Administration group was given by Tracy Di Marco White: ``Members: Christos Zoulas , on call, AEC vice chair, Tracy Di Marco White , on call, AEC chair, Bill Squier , Jan Schaumann , Phil Nelson , www synchronisation, Scott Reynolds , AEC, Jonathan Perkin , on call Noriyuki Soda , on call Soren S. Jorvang , on call S.P.Zeidler , on call Jason Thorpe , Thor Lancelot Simon New hardware: * A new backup server locate at Ludd, University of Lulea, Sweden, maintained by Anders Magnusson has been set up, and Thor has just started doing backups of everything we can fit onto it. * New http, console, mail, and one xen box that is used for administering the other Project servers. Thor built the system disks, and Erik Berls & Eric Volpe assisted in the installation of our new machines, as well as help from Peter Losher of ISC. * Two new build boxes were put together by Thor, and are currently on their way to California for Jason to install. * Sun has donated two machines for use in doing Solaris pkgsrc bulk builds, hosted at Stevens Institute of Technology, where Jan Schaumann maintains them. Hardware failures: This has been a bad year for us, at least regarding our hardware. We'd like to thank everyone for their patience during our outages. * The drives failed on the new http machine shortly after it went into service, causing me to do a fairly quick rebuild onto the new mail server hardware. Not long after that, the drives in the NEW new http server also showed some errors. Thor ordered new raptor SATA drives to replace these drives, and Jason put them into service. With the new drives in place, we should be able to move the mail server to the new hardware in the near future. While www.NetBSD.org was unavailable, Manuel Bouyer made www.fr.NetBSD.org available to answer www.NetBSD.org. Soda recovered mail lost because mail-index was unavailable, and Soren refed everything into mail-index. * The release engineering tender machine (releng) had a power supply / motherboard failure. Todd Whitesel picked up the machine to investigate the problems and how to repair it, but was unable to make it boot. I moved the release engineering ticket tracking service (req) to www.NetBSD.org, with help from Jim Wise, after Erik Berls mounted the hard drives and put the data up for me to use to reinstall it. * The release engineering build machine failed to successfully build various 2.0 builds due to various hardware failures. We were unable to use it for the 2.0 release sets. * The ftp server's raid controller failed multiple drives, causing Thor & I several days of pain while we backed up what we could and rebuilt and reinstalled it. Peter Losher and Eric Volpe were immense help juggling disks and installing new disks. The multiple disks failures were traced to a bug in the raid controller firmware. Many people were involved in this, if I've forgotten anyone I apologize. * The ftp server suffered from some frequent crashes that were traced to memory errors, solved by reseating the memory, and upgrading the bios. * The cvs server failed a disk, but after testing we determined the disk was actually ok. The machine was able to rebuild onto a hot spare for the duration. * The anoncvs server suffered a raid controller failure that manifested as single bit errors occurring frequently in our publically available source trees. After significant investigation, we traced this to the raid controller, and Thor provided a replacement raid controller. Petra, our newest admin, had this fail in her first on call shift, and worked quickly to make available update from the ftp server that are normally only available from the anoncvs server. For all of our hardware failures, Peter Losher of ISC provided invaluable help, swapping hardware and escorting our helpers in to do hands on work. Changes: * We have simplified and corrected the trust relationships between the Project servers. [...] * We have begun to migrate towards a standard hardware and software package for Project servers, replacing many old 4U machines with non-redundant disks and random old versions of NetBSD with modern 1/2U servers running 2.0. We have decreased our rack usage by more than 10U. Thor has done the majority of the design work involved, and Jonathan has started working on a system to ease package rebuilds for all of our Project servers. * We have established an on-call system for handling day to day admin tasks, and added new admins to the on call rotation. We are working to make sure issues brought to our attention don't fall through the cracks. * Soren Jorvang performed our migration from qmail to postfix, with some help from scripts Christoph Badura wrote after studying our qmail/majordomo configuration. * Soren also performed the migration from gnats3 to gnats4, and in the process also moved gnats from the mail server to the web server, allowing real time access to the gnats database from the web. * I implemented greylisting (light) with blacklisting under qmail this year. This caused some delay, but was widely appreciated given the amount of spam everyone was receiving through our mail host. On moving to postfix, Soren implemented greylisting using gld, which has caused much more comment, but has done an even better job of making our mailing list owners lives reasonable. We're currently testing a lighter version of gld. News: * We've established our own contract with 200 Paul (www.e200paul.com), so we can have our own contact able to physically work with the ISC hosted systems without being escorted by someone from ISC. Jason Thorpe has agreed to be our hands, and we deeply appreciate his ongoing help in resolving a problematic access situation that has persisted for years. This also allows us to directly ship hardware to 200 Paul. * ISC provided us with a /27 network after we ran out of room in the /28 we were previously using. Upcoming: * We will be implementing RT for use for security officer and for admin tasks. We are currently discussing the best way to do this. * Soren is looking into upgrading the mailing lists from majordomo to majordomo2. * Soren is also investigating moving mail-index from home grown scripts to mhonarc to facilitate easier access to the mailing list archives. Goals: * No broken machines! * Complete development and deployment of new backup system for all TNF hosts. * No broken machines! * Build on ongoing standardization of hardware and software configurations by deploying centralized configuration management. * No broken machines! Thank you for your patience through all our changes.'' * Closing After all the reports, Christos Zoulas thanked all the presenters for the interesting information and insight into the various project groups. If you are interested in participating in ongoing discussion, please join our mailing lists and see our web site for more information about the NetBSD project, http://www.NetBSD.org/. - Hubert Feyrer 2nd Chair of the Communications Executive Committee The NetBSD Foundation