OpenBSD/sun4v: porting OpenBSD to UltraSPARC T1 and T2

The SPARC architecture is a design promoted by Sun through an organisation that licences the design to other manufacturers. This is similar to the way the ARM architecture is managed. The SPARC architecture started at version 7 in 1986, in 32-bit mode. Version 8 was specified in 1990. Lots of implementations, from several vendors (Sun as well as others) were available on the market.

In 1993, version 9 was released. It introduced in particular 64-bit mode. The main vendors are Fujitsu with sparc64, and Sun itself with its UltraSPARC sun4u.

Unfortunately, the architecture is not completely specified, in particular the privileged mode (especially the MMU). They are large differences between the Fujitsu and the Sun chips, and both companies attempted to make them compatible: Fujitsu wanted Solaris to run on its chips. The result was the sun4v chip.

The K computer in Japan, the fastest in terms of LINPACK benchmark, runs on Fujitsu's SPARC64.

This is a RISC design, as opposed to the x86 architecture everybody and his dog runs.

The CPU has register windows: on function entry you can set aside the whole set of 32 generic registers and get a fresh new set, then restore the previous window on function return. This allows for fast function calls. Of course only so many windows are available, and deep function call graphs will still require to save registers one the stack beyond a certain depth.

The CPU features fast trap handling. In addition, interrupts can be sorted by priority.

Lastly, its MMU relies on software handling of cache misses.¹

The architecture was designed from the groundup to offer massive parallelism. A single processor includes 4 to 8 cores, and each core can execute 4 or 8 threads. The core will preempt a thread when it sees a load from memory, and switch to the least recently ran, runnable thread while waiting for the memory fetch.

This allows for up to 64 active threads per chip. Recent designs allow up to 4 chips per machine.

The raw speed of individual cores is rather low, typically 1.6GHz. However, those tradeoffs were made in order to reduce power consumption, heat dissipation, and still achieve high efficienty. The K computer is 6th on the June 2011 Green500 list when measured in FLOPS per Watt.

Mark started from the sun4u OpenBSD port, with the goal of having a single kernel for the two architectures, sun4u and sun4v. It should be noted that Solaris uses two different kernels.

For bare metal booting, no change was necessary for the first and second stage of boot(8). However, the kernel itself needed a lot of changes. The approach Mark used was to patch the code at run-time to handle the differences.

The UltraSPARC CPU allows the creation of domains, that allow the administrator to split the machine into slices. The hypervisor is in the firmware (and as I understood it, it doesn't require a core for itself). Machine resources can be attributed to domains for their exclusive use: vCPUs (hardware threads), memory, I/O devices, cryptographic resources (there is hardware support for MD5, SHA, RSA and other cryptographic services, but fewer crypto units than cores). There is no overcommit of memory or CPU.

One domain is privilege: the control domain has access to the hypervisor, and can be used to configure the other domains, in particular resource partition.

Service domains present virtual devices to the other domains, arbitrating between guest domains. The control domain is typically also a service domain.

I/O domain have direct access to the hardware devices. Support for running in a I/O domain, talking directly to the NIC or the disk was added rather quickly to OpenBSD; but to make good use of the hardware, it is preferable to run as many services in a guest domain as possible.²

Guest domains only use virtual devices exposed by a service domain. Sun's recommendation, of course, is to run as few applications in service and I/O domains as possible.

The first step in porting OpenBSD to sun4v was to have it run as a guest domain.

An I/O domain was created from Solaris as a control domain. In this I/O domain, OpenBSD was booted diskless (root over NFS), from a real dedicated NIC. Once this was done, hacking on vnet(4), the device driver for the virtual NIC provided by a service domain, started. Once vnet(4) was available, it was possible to start OpenBSD diskless, this time using the virtual NIC instead of a dedicated one.

At this point, OpenBSD can run in a guest domain, from an NFS root accessed through vnet(4). Hacking on vdsk(4) to access a virtual disk provided by a service domain could now start. After some hacking, OpenBSD could access (and boot from) a vdsk(4) device.

A communication system between domains exists. It consists of 64-byte messages. Sun makes no promises on the reliability of theses messages. However, Mark never noticed any message loss. He conjectured that Sun did not guarantee the reliability of the messages in order to be able to implement message passing over the network between domains running on separate hosts, maybe even to support hot migration of guest domains across physical hosts.

Be it as it may, Sun has defined reliable data streams over those 64-byte messages (called logical domain channels). The virtual NIC and disk devices implemented in OpenBSD follow the Sun specified protocol.

All features of the control domain are not available to OpenBSD for the moment. In particular, configuring domains does not work. Initial suport for starting and stopping other domains exists, but isn't ready yet. It is a good idea to keep a Solaris disk around for the moment.

OpenBSD is able to run as a service domain, as an I/O domain, and as a guest domain, servicing a Solaris guest domain or using virtual devices exported by a Solaris service domain. The specific setup Mark had in mind was using OpenBSD as a service domain with pf running on the real hardware, protecting several (Solaris or OpenBSD) guest domains.