NetBSD 10: 3 Years in the Making

Nia Alarie, EuroBSDCon 2022

  \-__,------,___.
   \        __,---`  NetBSD/evbarm efiboot (arm64)
    \       `---,_.  Revision 2.10 (Thu Aug 19 12:13:37 UTC 2021)
     \-,_____,.---`  Memory: 512 MB
      \
       \
        \

Release flow (not to scale!)

HEAD------------------------------------------------------------[...]
              |                                         |
              |                                         |
              |                                         |
              |                                      netbsd-10 ---|
              |                                                  10.0
              |
              netbsd-9 ------|----------|---------|-------------[...]
                            9.0        9.1       9.2

[...]--|--------------------------|--------------------fin
        8.1                        8.2

Quality assurance tools

                         build.sh
                            |
                            |
      ----------------------|
     /                      |
Static analysis             |
(-Werror, lint, Coverity...)|
                            |
                          Anita
             (9+ archs - QEMU/NVMM, gexmul...)
                            |
                       ------------------  
                      /                  \
                 Rump kernels      Sanitizers
                      |            (KUBSAN, KASAN, KMSAN...)
                      |            and KASSERT()
                     ATF                 |
                10,000+ tests            |
                                   Kernel fuzzers (syzbot)

Late 2019

Andrew Doran begins reworking critical subsystems of the NetBSD kernel.

scheduler: Improve performance and interactivity. [ad 20191201]
vfs: Reduce false sharing and lock overhead during normal operation.
	[ad 20191201]
uvm: Replace global counters with per-CPU counters, significantly
	reducing system time on workloads that are VM-system heavy.
	[ad 20191216]
scheduler: Make the scheduler topology aware.  It understands and
	makes good use of HyperThreading/SMT, spreads the load evenly
	across different physical CPU packages, and can effectively
	run asymmetric systems with slow and fast CPUs [ad 20200113]

By November 2020, performance on multi-core systems almost doubles.

Trouble a'brewing

current-users@, early 2020:

'diagnostic assertion "uvm_page_owner_locked_p(new_pg, false)" failed'
'panic: _pmap_enter: cannot allocate L3 table (aarch64)'
... and other fun kernel panics

Many bugs found with syzbot, then fixed.

Lessons learned?

For finding and fixing bugs early?

User participation in testing the unstable branch (current-users@) critical
Kernel fuzzers (syzbot) and sanitizers critical
Use of KASSERT() in kernel code critical

January 2020 - Great Driver Cleanup

Want wide improvement to kernel APIs / networking stack.
Some non-Ethernet networks fallen out of use entirely.
Feature removals in NetBSD quite rare.

Driver Cleanup Results

Gone: all FDDI, all HIPPI, all Token Ring, SMBFS, strip(4), de(4), uyurex(4), uyap(4)

Kept: sl(4) (serial line IP), se(4) (SCSI Ethernet adapter), pf(4) (the deprecated packet filter)

It went well compared to previous removal proposals!

Summer 2020

"We're all excited about the fabulous progress that -current has made, especially in performance but it's also quite obvious that it has serious issues and we are far from being able to branch netbsd-10"

Summer 2020: random saga begins

Early 2020: Entropy estimator removed from kernel - machines with trusted RNG block never, others until manual intervention.
Some hardware lacks any good random source - NetBSD supports a lot of hardware not built for modern encryption needs.
Disagreements about how to "trust" randomness, whether blocking or never blocking is okay.
getrandom adopted from Linux due to compromise behavior.
See Taylor's EuroBSDCon 2021 talk...

Unforseen problems: "Entropy error blocks lang/python38 installation"

2020 in Summary

New AArch64 security features (Pointer Authentication, Branch Target Identification)
Massive improvements to Xen support (PVH, fast MPSAFE networking)
ARM EFI bootloader improvements (boot.cfg, modules...)
Raspberry Pi 4 support
- ... with EDK II UEFI firmware required
wg(4) imported
- Mr. Donenfeld displeased
Retrocomputing: Major SMP improvements on Alpha
Default package database location moved...
- Rise of the "split brain" problem!

May 2021: CFARGS audit and kernel APIs

NetBSD 9.x:

Pile of functions: config_found, config_found_sm_loc, config_found_ia...
State of kernel device configuration: some code copy and pasted, some code misusing functions.

Verdict:

Simplify writing device configuration code for the NetBSD kernel
Challenge: NetBSD 9.x had (some) level of compile-time error checking. Preserve this and make it easier to use.

August 2021: CFARGS solution

Old example:

    c->c_dev = config_found(sc->sc_dev, &pa, pciprint,
        CFARG_SUBMATCH, config_stdsubmatch,
        CFARG_LOCATORS, locs,
        CFARG_DEVHANDLE, devhandle,
        CFARG_EOL);

New example:

    c->c_dev = config_found(sc->sc_dev, &pa, pciprint, 
        CFARGS(.submatch = config_stdsubmatch,
               .locators = locs, 
               .devhandle = devhandle));

DRM update

Long term problem: Users sad about our 4-year-old GPU support, lack of AMDGPU.
Riastradh to current-users@: "HEADS UP: Merging drm update"
GPU drivers synced with Linux 5.6, much newer hardware support.

DRM update challenges

Some errors difficult to debug (e.g. "strange colors on screen").
Many users interested testing the branch before merge without the technical skills.
Users confused at our priority to update without regressions ("where's amdgpu?")
Matching Linux semantics difficult.
Lack of assertions in code ported from Linux.

Once again, KASSERT critical to get useful information from users for debugging...

How can we help users test large kernel changes?

2021 in Summary

arm: Support for Apple M1, Amlogic G12 (ODROID-N2+), Rockchip RK3288, Allwinner V3s, performance improvements...
compat_linux(8): add various new system calls, support AArch64
Major VirtIO improvements (support for newer device standards)
regex: Add NLS support and gnu regex extensions (off by default). Some GNU regexes that were accepted before are no longer!
Various kernel API audits completed, many bugs found and fixed
New password hashing and disk encryption keygen method (Argon2id)
Start of UDF file system improvements - fsck_udf(8) by 2022
GPU driver update, synced with Linux 5.6.
Switched to GCC 10

Early 2022

Rush of changes to kernel interfaces before they have to freeze for -10 - we keep kernel modules compatible within a stable branch.
opencrypto(9) cleanup, usbenet(9) overhaul, preparation for making more kernel data structures opaque...
Version reaching as high as 9.99.99 - never been done before

Lessons learned from pkgsrc:

A rush before freezing for a new branch just delays the branch ;)

Mid 2022

Foot placed down by releng@.

File system corruption after NetBSD 10 file system shared with NetBSD 9.
"No more major changes until FFS ACL compatibility issue resolved"
"FFSv2ea" file system type introduced as a fix.
Mailing list thread

Unfinished business

Wi-Fi renewal - we need more volunteers/hardware
PF deprecation
DRM update - again?

Conclusions

NetBSD 10.0:

Massively better performance (especially on multi-processor systems).
Device drivers: 19+ new, 13+ extended, 13+ removed. Support for new GPUs, Apple M1, Raspberry Pi 4, PINE64 Quartz64, ODROID-N2+, Asus Tinker Board, VMware ESXI-Arm, Xen PVH...
Userspace programs: 5+ new (aiomixer, realpath, ioctlprint...), 20+ significantly improved (sh, script, vmstat, inetd...), 1 removed.
2000+ new test cases, kernel memory sanitizer, kernel concurrency sanitizer, various kernel-wide audits.
Compatibillity with WireGuard, faster better disk encryption, new ARMv8 security features.
Enhanced UDF filesystem, compat_linux(8), FFS access control lists, newer FUSE...

WORTH IT. It's been an amazing journey! Thanks everyone!