(Tool-)chaining the Hydra

The ongoing quest for modern toolchains in NetBSD

Martin Husemann & Jörg Sonnenberger


  • NetBSD has always been very conservative with compiler updates, since we have a huge zoo to feed (and verify new compilers work with)
  • The only modern toolchain in tree early last year, however, was clang, and it only worked on very few of our architectures.
  • A "gcc modernizing" campaign was started, and clang updated several times, which brought support for more architectures.


Besides the compiler/toolchain work, a big overhaul of the supporting cast (compiler runtime libraries, C startup, exception handling) was done,

So untill very recently, NetBSD had three gcc versions (4.1, 4.5, 4.8), one pcc and one clang version in the main source tree. Luckily gcc 4.1 is now gone (VAX switched to gcc 4.8 instead).

Still, this is not enough: one architecture needs newer gcc/binutils from pkgsrc.


There is a whole team involved in this work, as it is impossible to do without access to various test machines, people debugging on various architectures and makefile/gcc build system wizards.

The people working on it are:

  • Matt Thomas
  • Christos Zoulas
  • matthew green
  • Nick Hudson
  • Tohru Nishimura
  • Frank Wille
  • Yours truly
  • ... (sorry if we forgot mentioning you)

The Problem Matrix

  • The official build for a NetBSD-current release consists of 70 different sets of binaries
  • Sets for at least one architecture are not included in that set, because of open toolchain problems
  • Even ignoring the specific machine specific variants, the list of build architectures has 39 entries
  • For some of these architectures, multiple compiler options are available
  • For some machines, some of the working compilers are too big to be run natively

The Full List

The Missing One

But the above list is missing the playstation2 build, since the MIPS r5900 CPU used in that requires a newer binutils than we have in tree, and especially at least gcc 4.9.

It requires setting EXTERNAL_TOOLCHAIN and can not easily be build on the build cluster.

Note that the playstation2 port is not fully working (again) yet - so this is no high priority issue.

Runtime Restrictions

The SUN2 port only supports static linking and has strict virtual address space limitations (in addition to typical machines maxing out at 8 MB).

We did not manage to make gcc linkable for that machine yet

The SUN3 port supports shared libraries and we can build gcc 4.8 for it, but with typical machines having 16 MB maximum RAM, at (native) runtime gcc is only good enough to compile "hello world" type of programs.

The SUN3 problem already existed with gcc 4.5 and the last NetBSD release, so strictly it is no regression.

Possible Solution

We hope pcc to become usable as a native compiler on this extremely restricted machines.

Anders Magnusson has added m68k support and is looking at it.

We might ship SUN2 and SUN3 releases (cross-)compiled with gcc 4.8 but delivering pcc as a native compiler in the future.

(And ignore that this is not a very usefull environment to build Firefox or random stuff from pkgsrc due to lack of C++ support)

Licenses Allowed

NetBSD, just as all other BSDs, is strongly trying to get rid of all non BSD licensed stuff in the base system.

However, we also try to supply a fully usable system, and we want to be able to cross-compile all of this automatically on any posixishly host.

Board decided to allow inclusion of GPLv3 licensed code, but with strict restrictions:

  • (Basically) Restricted to toolchain
    (That is stuff you do not need at runtime in embedded products for example)
  • Build controlled by make variable
    (easy to build the system without)
  • Living in a special directory: src/external/gpl3
    (so easy to remove from the soure tree)

License Separation

Triggered by this GPLv3 source organization decision, we started cleaning up other stuff per license as well:


Of course it is all still buildable from the standard source tree, via reachover makefiles!

Compiler selection

License GPL3 BSD BSD
Platforms (mature) All x86, ARM x86
Mostly working PPC, SPARC M68K, VAX
Pro Good Optimiser Good Optimiser
Integrated assembler
Small & very fast
Contra Full rebuild for each target. C++11 Weak optimizer

Towards a full BSD runtime

crtstuff ⇒ rewritten

libgcc(_s) ⇒ compiler_rt and libunwind in libc.

libstdc++/libsupc++ ⇒ libc++ including libc++rt


Provides support routines for operations not implemented by the hardware.

Typical example: software float, division

Optimised assembler code for ARM exist, other platforms desirable.

Special case: milli code ⇒ functions embedded in each shared library.

Open issue: division on SH3 uses non-standard call ABI.

Open issue: symbol versioning fallout for "empty" libgcc_s.so.

C++ support

libc++ provides a modern C++11 and C++14 runtime, optimised for speed.

Code is as readable as template heavy C++ code can be.

Many easy-to-run test cases.

libc++rt for low-level support as (re)licensed from Pathscale by TNF and FreeBSD Foundation.

Other languages

OpenMP: undecided whether we will ship Intel's runtime in the future.

libobjc: free implementation from David Chisnall exists, currently not in base.

libgfortran: no plan for shipping.

Exception handling

C++ is the main consumer, but open for other languages like Java.

General process:

  • Save current thread state.
  • Unwind the stack to find call regions.
  • Look for exception handlers based on Program Counter.
  • On match: execute and either clean-up or continue.

Useful for backtraces as well. — C++ in the kernel

Unwinding in NetBSD

Based on the Itanium C++ ABI with platform specific variances.

Based on Apple's libunwind with heavy modifications.

Only provide the _Unwind_* interface, not the HP unw_* API.


Executive summary: NIH

Semantically different from Itanium C++ ABI.

Different annotation from all other platforms.

Difficult to strip unwind data off when not needed.

More compact -- but so is Apple's Compact Unwind format.

Let's not do that!

ARM with DWARF unwind: 20 lines patch for GCC, 140 lines for libunwind

Unwinding on SPARC

Platform uses register windows: 8 output registers become input register of called function. Large register file on chip, kernel support for flushing oldest entries to RAM on overflow.

Special DWARF instruction to facilate this. — Very compact FDEs!

Return address is not the instruction after the call.

Unwinding on HPPA

Return address contains privilege level in the lowest two bits.

Not a fixed value when dealing with kernel backtraces!

Unwinding on VAX

callg and calls are epitomes of CISC design:

  • Automatic stack alignment
  • Function describes which registers to save and restore.
  • Clean-up arguments from the stack on return.
  • Clean-up size depends on the caller

Special DWARF instruction for stack-clean up needed.

Lazy binding introduces intermediate stack frame

...but we try to avoid it by rerunning the call after binding.

GCC 4.5 and 4.8 create bogus DWARF by default.