bhyvecon Tokyo 2019 - The BSD Hypervisor Conference

Author: Kamil Rytarowski

E-mail: kamil@netbsd.org

Date: March 20th, 2019

Location: Tokyo, Japan

Kamil Rytarowski (born 1987)

Krakow, Poland

NetBSD user since 6.1.

NetBSD Foundation member since 2015.

Work areas: kernel, userland, pkgsrc.

Interest: NetBSD on desktop and in particular NetBSD as a workstation.

Current activity in 3rd party software:

• LLVM committer.
• GDB & binutils committer.
• NetBSD maintenance in qemu.

• Terminology
• NetBSD and software emulation
• NetBSD and hardware assisted emulation
• Future plans

Virtualization is a very broad term. We refer here to running a full Operating System [instance] (guest) on another already running Operating System [instance] (host).

There are two basic types of virtualization:

• Full software emulation.
• Hardware assisted emulation.

A hypervisor (Virtual Machine Monitor) is software, firmware, or hardware that creates and runs virtual machines.

There are two types of hypervisors:

• Type-1 (native/bare-metal hypervisors) - running directly on hardware.
• Type-2 (hosted hypervisors) - running as a kernel extension (module) or a userland application.

Software emulation VS hardware assisted emulation

Full software emulation is slower, but can support cross-CPU virtualization and is typically accessible without privileged hardware features.

Hardware assisted virtualization uses CPU specific features in order to get near native execution speed of a guest machine, but it's usually restricted to the same CPU architecture only and requires privileged kernel drivers.

There are two types of virtualization in the interest of NetBSD developers and users:

• NetBSD as a host virtualizing a given Operating System as guest.
• NetBSD as a guest virtualized on a given Operating System as host.

Whenever applicable and expected, it's desired to support NetBSD as both guest and host Operating System.

For practical reasons it's necessary to handle NetBSD as a guest Operating System in foreign environments (such as remote VPS services) and virtualize non-NetBSD systems on a NetBSD host (such as retro computing).

Does it still matter in NetBSD? Yes!

Software emulation matters in NetBSD in particular for:

• Development of new ports.
• Maintenance of existing ports.
• Featured kernel cross-CPU debugging and introspection.

NetBSD/riscv running on the Spike emulator

https://twitter.com/zmcgrew/status/1055682596812730368

http://releng.netbsd.org/test-results.html

Probably the most frequently used software virtual machines for NetBSD guests:

• bochs (i386)
• giano (emips, vax)
• gxemul (algor, arc, cats, cobalt, dreamcast, evbarm, evbmips, hpcmips, landisk, macppc, netwinder, pmax, prep, sgimips)
• QEMU (amd64, evbarm, i386, sparc, sparc64)
• simh (vax)
• simics (amd64)
• ski (ia64)
• spike (riscv)
• tme (sparc, sparc64, sun2, sun3)
• xm6i (x68k)

https://www.netbsd.org/ports/emulators.html

Software emulation is not good for everything, especially whenever we depend upon performance.

Performance and cost of execution are mandatory elements of practically all production work loads.

Whenever we require performance we need hardware assisted emulation.

Due to practical reasons and current resources of the project, the main focus of hardware assisted emulation is the x86 (Intel and AMD) architecture.

However, there is some activity and interest in the aarch64 land as well.

Currently available options:

• VM86 (removed in mid 2017)
• NetBSD/xen
• NVMM (NetBSD Virtual Machine Monitor)
• HAXM (Intel Hardware Accelerated eXecution Manager)

https://github.com/NetBSD/src/commit/5929a332ef850162327ea78d56ab9c1900daf71f

http://wiki.netbsd.org/ports/xen/

https://wiki.xen.org/wiki/Xen_Project_Software_Overview#Guest_Types

http://wiki.netbsd.org/ports/xen/howto/

Multiprocessor (SMP) support in NetBSD differs depending on the domain:

• dom0 - not supported (without experimental patches)
• domU - supported

The standard approach is to use NetBSD/amd64 for the dom0.

The dom0 system, plus each domU, can be either i386PAE or amd64. i386 without PAE is not supported.

For domUs, i386PAE is considered faster than amd64.

http://wiki.netbsd.org/ports/xen/howto/

There are two hardware accelerated engines in the NetBSD kernel.

HAXM (by Intel, ported to NetBSD by Kamil Rytarowski):

• Multiplatform virtualization engine (NetBSD, Linux, Windows, macOS)
• Designed for Intel CPUs and x86_64 ISA only (AMD SVM is unsupported)
• External open-source project from Intel available through pkgsrc
• Works with at least NetBSD 8.0, could work with older releases but not tested

NVMM (by Maxime Villard):

• Homegrown NetBSD virtualization engine
• Machine Independent API with NetBSD/amd64 support (Intel and AMD CPUs) as host
• Integral part of the NetBSD distribution
• Works with NetBSD-current and planned for future releases (starting with 9.0)

Both engines:

• are supported on NetBSD with QEMU (in versions from pkgsrc).
• require relatively recent CPUs, not older than a decade old hardware.
• designed to behave as a backend to QEMU or similar frontend only, without full-stack software virtualization.
• are designed to reuse drivers from QEMU.
• combined with QEMU can run at least NetBSD, Windows, Linux and several other Operating Systems in at least a single guest version/type (like 32/64 bit).
• can run multiple Virtual Machines with multiple VCPUs in each of them.
• are open-source and BSD-licensed.

Why are there two kernel engines for the same purpose?

• HAXM porting started much earlier than any NVMM work and targets the hardware of interest of Intel CPU users in desktop-workflows
• NVMM started initially as AMD SVM hypervisor and is currently designed to be machine independent (AMD SVM, Intel VTx, open for other CPUs)
• HAXM is a cross-OS hypervisor API, it's cheaper to port existing software using it to NetBSD, without (or before) remapping all the KVM APIs to NVMM
• HAXM works with older unmodified kernels including the latest stable release 8.0, NVMM is in development in NetBSD-current and planed for future releases

All in all: HAXM allows to get the job done earlier and reduce cost of porting software to NetBSD (expanding the QEMU support for NetBSD/HAXM was a 1-liner patch), NVMM is the long term recommendation (and requires more porting).

Could HAXM and NVMM be merged? No, as both have different targets and the legacy of HAXM APIs and design choices (with accumulated technical debt) restricts it to Intel x86 desktop users. It was easier to start from scratch than totally rewrite existing code.

http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge

http://m00nbsd.net/4e0798b7f2620c965d0dd9d6a7a2f296.html

There is in-kernel hypervisor detection and extra handling of VMware, Microsoft HyperV, Xen and KVM.

There are available special device drivers for virtualization:

• VirtualBox Guest Additions
• Hyper-V/Azure
• VMware VMXNET3 Virtual Interface Controller (vmx(4))
• Xen PV (xenbus(4), pciback(4), xbd(4), xbdback(4), xennet(4), xpci(4), xvif(4))
• virtio disk device (ld(4))
• virtio Ethernet device (vioif(4))
• virtio pseudo-device to release memory back to the hypervisor (viomb(4))
• virtio entropy source (viornd(4))
• virtio SCSI adapter (vioscsi(4))

Pending tasks:

• Address PaX MPROTECT (W^X) violation issues
• bsd-user - restore the support from scratch
• Upstream local pkgsrc patches (mostly old NetBSD and SmartOS)

Long term/research:

• HQEMU - LLVM code generation in software emulation
• device pass-through (PCI, USB, ...)
• DPDK

https://wiki.netbsd.org/users/kamil/qemu/

Enable NetBSD support for:

• dom0 SMP
• PVHVM
• PVH

If possible port Xen to NetBSD/evbarm and NetBSD/aarch64.

NVMM pending tasks:

• upstream QEMU support patches
• stabilize and address bugs
• NetBSD/aarch64 port

HAXM pending tasks:

• fill missing gap of features that are known to work better on other hosts
• whenever possible collaborate with upstream and help to address generic issues

• Test the NetBSD host support of Xen, HAXM and NVMM in your workflow.
• Get in touch with developers and the user community.
• Report bugs.
• Contribute patches.