NetBSD 10: 3 Years in the Making
Nia Alarie, EuroBSDCon 2022
Notes for this talk
\ __,---` NetBSD/evbarm efiboot (arm64)
\ `---,_. Revision 2.10 (Thu Aug 19 12:13:37 UTC 2021)
\-,_____,.---` Memory: 512 MB
Release flow (not to scale!)
| netbsd-10 ---|
9.0 9.1 9.2
Quality assurance tools
Static analysis |
(-Werror, lint, Coverity...)|
(9+ archs - QEMU/NVMM, gexmul...)
Rump kernels Sanitizers
| (KUBSAN, KASAN, KMSAN...)
| and KASSERT()
10,000+ tests |
Kernel fuzzers (syzbot)
Andrew Doran begins reworking critical subsystems of the NetBSD kernel.
scheduler: Improve performance and interactivity. [ad 20191201]
vfs: Reduce false sharing and lock overhead during normal operation.
uvm: Replace global counters with per-CPU counters, significantly
reducing system time on workloads that are VM-system heavy.
scheduler: Make the scheduler topology aware. It understands and
makes good use of HyperThreading/SMT, spreads the load evenly
across different physical CPU packages, and can effectively
run asymmetric systems with slow and fast CPUs [ad 20200113]
By November 2020, performance on multi-core systems almost doubles.
current-users@, early 2020:
- 'diagnostic assertion "uvm_page_owner_locked_p(new_pg, false)" failed'
- 'panic: _pmap_enter: cannot allocate L3 table (aarch64)'
- ... and other fun kernel panics
Many bugs found with syzbot, then fixed.
For finding and fixing bugs early?
User participation in testing the unstable branch (current-users@)
Kernel fuzzers (syzbot) and sanitizers critical
Use of KASSERT() in kernel code critical
January 2020 - Great Driver Cleanup
Want wide improvement to kernel APIs / networking stack.
Some non-Ethernet networks fallen out of use entirely.
Feature removals in NetBSD quite rare.
Driver Cleanup Results
Gone: all FDDI, all HIPPI, all Token Ring,
SMBFS, strip(4), de(4), uyurex(4), uyap(4)
Kept: sl(4) (serial line IP), se(4) (SCSI Ethernet adapter),
pf(4) (the deprecated packet filter)
It went well compared to previous removal proposals!
"We're all excited about the fabulous progress that -current has made,
especially in performance but it's also quite obvious that it has
serious issues and we are far from being able to branch netbsd-10"
Summer 2020: random saga begins
- Early 2020: Entropy estimator removed from kernel - machines with
trusted RNG block never, others until manual intervention.
- Some hardware lacks any good random source - NetBSD supports a
lot of hardware not built for modern encryption needs.
- Disagreements about how to "trust" randomness, whether blocking
or never blocking is okay.
- getrandom adopted from Linux due to compromise behavior.
- See Taylor's EuroBSDCon 2021 talk...
Unforseen problems: "Entropy error blocks lang/python38 installation"
2020 in Summary
- New AArch64 security features
(Pointer Authentication, Branch Target Identification)
- Massive improvements to Xen support (PVH, fast MPSAFE networking)
- ARM EFI bootloader improvements (boot.cfg, modules...)
- Raspberry Pi 4 support
- ... with EDK II UEFI firmware required
- wg(4) imported
- Retrocomputing: Major SMP improvements on Alpha
- Default package database location moved...
- Rise of the "split brain" problem!
May 2021: CFARGS audit and kernel APIs
- Pile of functions:
- State of kernel device configuration: some code copy and pasted,
some code misusing functions.
- Simplify writing device configuration code for the NetBSD kernel
- Challenge: NetBSD 9.x had (some) level of compile-time
error checking. Preserve this and make it easier to use.
August 2021: CFARGS solution
c->c_dev = config_found(sc->sc_dev, &pa, pciprint,
c->c_dev = config_found(sc->sc_dev, &pa, pciprint,
CFARGS(.submatch = config_stdsubmatch,
.locators = locs,
.devhandle = devhandle));
- Long term problem: Users sad about our 4-year-old GPU support,
lack of AMDGPU.
- Riastradh to current-users@: "HEADS UP: Merging drm update"
- GPU drivers synced with Linux 5.6, much newer hardware support.
DRM update challenges
- Some errors difficult to debug (e.g. "strange colors on screen").
- Many users interested testing the branch before merge without
the technical skills.
- Users confused at our priority to update without regressions
- Matching Linux semantics difficult.
- Lack of assertions in code ported from Linux.
Once again, KASSERT critical to get useful information from
users for debugging...
How can we help users test large kernel changes?
2021 in Summary
- arm: Support for Apple M1, Amlogic G12 (ODROID-N2+), Rockchip RK3288,
Allwinner V3s, performance improvements...
- compat_linux(8): add various new system calls, support AArch64
- Major VirtIO improvements (support for newer device standards)
- regex: Add NLS support and gnu regex extensions (off by default).
Some GNU regexes that were accepted before are no longer!
- Various kernel API audits completed, many bugs found and fixed
- New password hashing and disk encryption keygen method (Argon2id)
- Start of UDF file system improvements - fsck_udf(8) by 2022
- GPU driver update, synced with Linux 5.6.
- Switched to GCC 10
Rush of changes to kernel interfaces before they have
to freeze for -10 - we keep kernel modules compatible
within a stable branch.
opencrypto(9) cleanup, usbenet(9) overhaul, preparation for
making more kernel data structures opaque...
Version reaching as high as 9.99.99 - never been done before
Lessons learned from pkgsrc:
- A rush before freezing for a new branch just delays the
Foot placed down by releng@.
- File system corruption after NetBSD 10 file system shared
with NetBSD 9.
- "No more major changes until FFS ACL compatibility issue resolved"
- "FFSv2ea" file system type introduced as a fix.
- Mailing list thread
- Wi-Fi renewal - we need more volunteers/hardware
- PF deprecation
- DRM update - again?
- Massively better performance (especially on multi-processor systems).
- Device drivers: 19+ new, 13+ extended, 13+ removed.
Support for new GPUs, Apple M1, Raspberry Pi 4, PINE64 Quartz64,
ODROID-N2+, Asus Tinker Board, VMware ESXI-Arm, Xen PVH...
- Userspace programs: 5+ new (aiomixer, realpath, ioctlprint...),
20+ significantly improved (sh, script, vmstat, inetd...), 1 removed.
- 2000+ new test cases, kernel memory sanitizer, kernel concurrency
sanitizer, various kernel-wide audits.
- Compatibillity with WireGuard, faster better disk encryption,
new ARMv8 security features.
- Enhanced UDF filesystem, compat_linux(8), FFS access control lists,
WORTH IT. It's been an amazing journey! Thanks everyone!