Running Xen with a NetBSD Dom0

Update 2018-03-11: I have given up on NetBSD/Xen and now use Gentoo GNU/Linux/Xen instead. The reason is that I ran into stability problems which survived many NetBSD updates. I still use NetBSD in production, i.e., as firewall with Xen PCI passthrough.

Preliminaries

Getting Xen to run is not easy, at least it is not easy with a NetBSD DOM0 since the few guides out there are wrong and/or incomplete, including NetBSD's own This guide is complete and has been tested on a variety of systems.

Why not walk down the wider path, using GNU/Linux as DOM0? Well, if you like the kernel Linux, by all means, do that! I prefer an well-engineered kernel, so I choose NetBSD.

First we need to install NetBSD. A basic NetBSD install might be really easy, or really tricky, depending on various things. If you're using an old GPT-partitioned disk, expect to need manual steps where you blank out the disk. You might need a manual newfs command or two too. (An indication of this GPT-related problem is that the NetBSD installer fails to create partitions, claiming that it is already "open for reading" or some other non-sense error message.)

Unfortunately, NetBSD's installer now fails on many PCs from 2010 and later. There are several failure modes, the most common being that you get a prompter "root device: " where it hangs. No input helps. Other problems are crashes, or that the installer starts but the text is black on a black background.

Keep in mind that NetBSD is very competently maintained but also crude by design. Set up things 100% right, or expect hard-to-track-down problems. For example, for my NetBSD/Xen machines, I prefer not to let the installer put a full-blown X11 system on my lean /usr partition, and instead install needed pieces from source (using pkgsrc). If one, god forbid, forgets to put X11_TYPE=modular in /etc/mk.conf, pkgsrc cannot be used for building anything X11 related. It doesn't track dependencies, and gives an error message about /usr/X11R7 (which is an obsolete path for X11).

For Xen to work (at least well) AMD SVM or Intel VMX need to be enabled in BIOS. Some BIOSes have these on by default, some have them off. If you upgrade your BIOS, they might be turned back off. (I am not sure about other toggles that could help, e.g., IOMMU. If you know, please help me improve this text!)

Talking about upgrading BIOS. I tend to follow the idiom "if it ain't broke, don't fix it". But my experience is that Xen needs BIOS fixes to run stably. 5 out of my 6 Xen systems were unstable under Xen first, but all became stable with a new, shining BIOS. They all had run stably under FreeBSD and/or NetBSD before I installed Xen and thus before I upgraded the BIOS.


Installing basic OS

We start with a NetBSD install. Get install media. I recommend using a late snapshot. Use any of the links, then grab an USB image images/NetBSD-7.0_STABLE-amd64-install.img.gz. Uncompress it, and put it on a USB stick using plain old dd. Then boot the installation program from this stick.

I recommend these snapshots as earlier NetBSD kernels have a serious bug in Xen scheduling which will make DomU's useless as soon as CPU usage in Dom0 goes up.

I do not intend this as a NetBSD install guide, I just point out some aspects that are relevant for getting Xen to work.

Note that the NetBSD installer first asks for all partition sizes, then in a later screen lets you set parameters for the partitions.

If you intend to boot with grub, change the partition type of / from the default FFSv2 to FFSv1. The other partitions can be either type, but the default FFSv2 is generally better.

Choose to install pkgsrc when asked. If you forget to do this, look at the NetBSD pkgsrc docs how to post-install it manually.

Finish the installation, reboot into your new system.


Installing Xen stuff

Install needed packages:

(cd /usr/pkgsrc/sysutils/xenkernel45; make install clean clean-depends)
(cd /usr/pkgsrc/sysutils/xentools45;  make install clean clean-depends)

(There is now also xenkernel46/xentools46. I haven't yet tried those, my using of the '45 variants is not a recommendation against '46.)


Download the NetBSD DOM0 kernel. Then do:

gunzip netbsd-XEN3_DOM0.gz
mv netbsd-XEN3_DOM0 /
(Don't get confused by the references to XEN3 here; this kernel works with Xen 4.2.)


Copy the Xen kernel in place (yes, the gz file) and Xen startup files where the NetBSD boot code finds them:

cp -p /usr/pkg/xen45-kernel/xen.gz /
cp -p /usr/pkg/share/examples/rc.d/xen* /etc/rc.d


Create needed Xen-related device files:

(cd /dev; ./MAKEDEV xen)
(This creates devices for just 4 guests; use
for i in `seq 4 20`; do (cd /dev; ./MAKEDEV vnd$i); done
to make a few more.)


Append to /etc/rc.conf:

xend=yes
xencommons=yes


Edit /boot.cfg. Here is the file I use:

menu=Boot normally:rndseed /var/db/entropy-file;boot netbsd
menu=Boot single user:rndseed /var/db/entropy-file;boot netbsd -s
menu=Disable ACPI:rndseed /var/db/entropy-file;boot netbsd -2
menu=Disable ACPI and SMP:rndseed /var/db/entropy-file;boot netbsd -12
menu=Boot Xen 1:load /netbsd-XEN3_DOM0 console=pc; multiboot /xen.gz dom0_mem=1024M
menu=Drop to boot prompt:prompt
default=5
timeout=5
clear=1
(Again, don't get confused by the references to XEN3 here; this kernel works with Xen 4.x.)

A reboot here should work, except for networking of emulated systems.


Networking

The right approach depends on whether the emulated systems will have private (such as 10/8, 192.168/16) or public IP addresses. Or more specifically, if the "upstream" interface of the host is not in the same private/public category as the emulated systems interfaces.

If the upstream host interfaces is public and the emulated systems interfaces are private, then do:

  1. Tell the boot sequence to configure just listed interfaces, and to configure tap0 before bridge0. Put these lines in /etc/rc.conf:
    auto_ifconfig=NO
    net_interfaces="`/sbin/ifconfig -l` tap0 bridge0"
    
  2. Create a tap0 interface and assign it e.g., 192.168.0.1. Put this in /etc/rc.conf:
    ifconfig_tap0="192.168.0.1 netmask 255.255.0.0"
    
  3. Create /etc/ifconfig.bridge0 with this contents:
    create
    !brconfig $int add tap0 up
    
  4. In order to allow IP routing between the tap interfaces connected to the bridge, append this to /etc/sysctl.conf:
    net.inet.ip.forwarding=1
    
  5. Probably setup a DHCP server for assigning IP addresses to emulated systems. The right interface to use for the DHCP server is tap0.
  6. If you wish emulated systems to access the Internet, setup a NAT, e.g., put
    ext_if=XXX
    ext_addr=X.Y.Z.T
    nat   on $ext_if from 192.168.0.0/16  to any -> $ext_addr
    
    in /etc/pf.conf and set pf=YES in /etc/rc.conf. Replace XXX with the host Ethernet interface of choice. Replace X.Y.Z.T with its IP address.
Else if the upstream host interface and the emulated systems' interfaces are either both private or both public, we can simply bridge the external network and our internal network. We will rely on some external DHCP server to give IP addressed to emulated systems. Do:
  1. Create /etc/ifconfig.bridge0 with this contents:
    create
    !brconfig $int add XXX up
    
    Replace XXX with the host Ethernet interface of choice.

Done! A reboot command should now bring up the new Xen+NetBSD system with functional networking, including network bridging for guests. If all is well, xm li should list the guests. It should look something like this now:
Name                  ID  Mem VCPUs    State   time(s)
Domain-0

Time for some guests! It is beyond this text to describe how to do that. Important note: Do NOT use qemu-img-xen for making guest images, or else dd with the seek options, or any other way of creating sparse files. Sparse files and vnd are incompatible in NetBSD and using sparse files will cause the system to be quite instable. You might not see this problem with small guest images, but as images get larger, you WILL be out of luck.