Hey EuroBSDCon - I'm nia. I'm on the Board of Directors of the NetBSD Foundation, maintain programs in the base OS and also in pkgsrc, do embedded and desktop stuff (particularly audio production). In this talk, I'm going to tell the story of NetBSD 10 - it's gonna go into the gritty details of release engineering, the wins and pitfalls, some of the technical and the community side. --- This lovely ASCII diagram shows how NetBSD releases are supported. What you can see is that we support two stable "branches" at a time, which get continuous updates, for example netbsd-8 and netbsd-9. When netbsd-10 is released, netbsd-10 and netbsd-9 become the supported branches, and netbsd-8 stops being supported. The so-called current branch is changing constantly, and might include breaking changes. A new stable branch is made when the current branch is deemed ready by the release engineering team. Changes are backported from current to the stable branches when they're deemed non-intrusive enough. Ocasionally, we make a release, which is a fixed snapshot of a stable branch. Releases don't get updates, so you should pay close attention to security advisories, and figure out whether it's necessary for you to update to a stable snapshot if you're using a release, or whether you can or should take other countermeasures. --- This next diagram shows some of the major tools we use for ensuring NetBSD branches are stable. build.sh is the main tool used for building NetBSD from the source code, it supports easy cross-compilation across many platforms and CPU architectures. In tandem with the builds, we use lots of static checks which cause clear immediate failures if the source code contains anything problematic. We've been doing this for over 20 years now. A Python script called Anita is a really important part of release engineering - it helps automatically install NetBSD in virtual machines for over 9 CPU architectures. This runs separately to tests on real hardware, which might be slow and awkward. Inside the virtual or physical machine, the NetBSD test suite containing over ten thousand tests runs. Through Rump, it runs various parts of the kernel in userspace, making tests very reliable and allowing us to simulate real hardware setups. We're also using various sanitizers, which detect common programming errors in the kernel at runtime, and feed various fuzzing tools which feed randomized data into the kernel to detect bugs. By now, it should be very clear that this is "state of the art", and very much needed for our small-ish team to maintain a large code base that's over 30 years old now. --- Late 2019 is where NetBSD 10.0 really starts to take form as something special. While the first release from the netbsd-9 branch is being stabilized, Andrew Doran starts making frantic commits that drasticaly improve the performance of the NetBSD CURRENT kernel, especially on multi-core systems, and those with a mixture of low-power and high-power cores like modern Intel and ARM processors. --- BUT... it immediately becomes clear that some of Andrew's changes are causing instability. Our tools instantly detect new failure cases in the kernel. Andrew now has to dedicate his time to ironing out and fixing them. --- What we learned very quickly is that user participation in testing is critical. A lot of kernel errors are detected at run-time, during operation, so it's good that we can get lots of detailed reports from users about our error checks failing. KASSERT is a tool that allows the kernel itself to indicate its state is wrong or confused, and report a message useful for developers, and it was critical in this time. --- In January 2020, some developers wanting to make networking on NetBSD faster discover some problems with older device drivers, for network technology that's fallen out of use. We've got a culture that is fairly against large feature removals, so we've got to tread carefully. For every single one, the user community is asked separately - is anyone using this hardware? --- So, in NetBSD 10.0 we're removing an unprecedented amount of code for old wired network technologies which were replaced by Gigabit Ethernet. Some are still useful to the retrocomputing community and were kept - NetBSD is a group of people with a range of interests, and it's important that we keep everyone relatively happy and avoid major conflicts. This was done really well on a social level. --- As we move through 2020, it becomes pretty clear to us all that NetBSD 10.0 is gonna be an amazing release, but we're still really far from being able to call it stable. Our release engineering and testing tools make that much obvious. --- By now it's Summer 2020. A lot of what I'm going to discuss here is covered in Taylor Campbell's EuroBSDCon talk from 2021, so I'm not going to focus on it too much. Please listen to that later. Fundamental to NetBSD is the fact we support lots of old, small, and crappy hardware, and some really new and fancy hardware. Some of this hardware is very suitable for modern Internet security, but some isn't. Specifically, really old or embedded hardware might lack any good sources of random data, important to Internet security, but it's plentiful on large x86 boxes. We needed to provide a good user experience in both cases. It took more than a year, and many flame wars in the community (my hands are not clean of this), for us to figure out how we wanted things to look. POSIX did make some decisions on our behalf. This is a great example of changes having unforseen consequences. --- And this is NetBSD 10.0's 2020 in summary -- we gained loads more security and encryption features, huge performance improvements for Xen virtual machines, as well as enhanced support for 64-bit ARM and alpha hardware. Take a look and read, it's all really nice stuff. A minor footnote, more on the pkgsrc side, is a change concerning the location of the package database. This caused unintended breakage on various people's machines, and a long discusison about how to fix it. --- It's now 2021, and a few developers are starting to focus on some potential wide improvements to the NetBSD kernel's code quality. It should be pretty obvious to most experienced programmers that an API that's not simple to use will result in lots of code being copy and pasted badly, and this was the case for config_found in some device drivers. The obvious solution is to make config_found easier for programmers to use, which was done. But what's more complicated is making sure it can't be misused even before the code is built. NetBSD 9 had some of this for config_found, and various developers objected to the simplification because it lost some of the error checking. A long discussion started, before an agreeable compromise was reached. --- And here's the result that was eventually settled on. Much prettier code, don't you think? --- By 2021, some of our users are starting to complain about our GPU support being limited to hardware that's a couple of years old by now. They're lucky -- a major graphics driver update was being worked on, with our code being synced with Linux 5.6, but with this comes thousands of lines of code changing, all at once. It's not going to be easy. --- Some challenges appeared quickly. Some problems are difficult to figure out, even with our kernel error checking tools - trippy colours appearing on the screen being a good example. A lot of users were interested in testing the update, but didn't have the knowledge required to report problems well. They were also confused by our priorities - updating existing working code and fixing bugs there before adding any new hardware support. KASSERT was very useful for getting diagnostic information from the kernel, but code ported from Linux doesn't necessarily use any equivalent. It was a problem. I would love to hear about ideas for making large kernel changes easier to test for inexperienced users. --- And here was our 2021 in summary. Again, we had huge improvements to support for 64-bit ARM, including big-ticket items like support for the Apple M1 processor. There were also improvements to Linux compatibility, and many bugs were found and fixed in the kernel. The UDF file system, a very interesting universal file system for portable drives and discs, saw major improvements. --- By early 2022, it was getting clear that we wanted to do a release soon. Unfortunately, when developers hear that there's going to be a release, they've got a tendency to rush their changes in. This adds more instability to the equation, and tends to delay the release even more. To make matters worse, we were running out of version numbers for CURRENT, which caused developers to group large changes together! It would be really nice to figure out an effective way to avoid this problem in the community. --- And we're getting to the end of our story, so this is our NetBSD 10.0 in summary. We've got massive performance improvements, leaps and bounds better hardware support, major improvements to testing, quality assurance, and our various user space programs, new security features, and a few greatly enhanced file systems. Was it worth the wait? You can be the judge of that!