Update 2018-03-11: I have given up on NetBSD/Xen and now use Gentoo GNU/Linux/Xen instead. The reason is that I ran into stability problems which survived many NetBSD updates. I still use NetBSD in production, i.e., as firewall with Xen PCI passthrough.
Getting Xen to run is not easy, at least it is not easy with a NetBSD DOM0 since the few guides out there are wrong and/or incomplete, including NetBSD's own This guide is complete and has been tested on a variety of systems.
Why not walk down the wider path, using GNU/Linux as DOM0? Well, if you like the kernel Linux, by all means, do that! I prefer an well-engineered kernel, so I choose NetBSD.
First we need to install NetBSD. A basic NetBSD install might be really easy, or really tricky, depending on various things. If you're using an old GPT-partitioned disk, expect to need manual steps where you blank out the disk. You might need a manual newfs command or two too. (An indication of this GPT-related problem is that the NetBSD installer fails to create partitions, claiming that it is already "open for reading" or some other non-sense error message.)
Unfortunately, NetBSD's installer now fails on many PCs from 2010 and later. There are several failure modes, the most common being that you get a prompter "root device: " where it hangs. No input helps. Other problems are crashes, or that the installer starts but the text is black on a black background.
Keep in mind that NetBSD is very competently maintained but also crude by
design. Set up things 100% right, or expect hard-to-track-down problems. For
example, for my NetBSD/Xen machines, I prefer not to let the installer put a
full-blown X11 system on my lean
/usr partition, and instead install needed pieces from source
(using pkgsrc). If one, god forbid, forgets to put
/etc/mk.conf, pkgsrc cannot be
used for building anything X11 related. It doesn't track dependencies, and
gives an error message about /usr/X11R7 (which is an obsolete path for X11).
For Xen to work (at least well) AMD SVM or Intel VMX need to be enabled in BIOS. Some BIOSes have these on by default, some have them off. If you upgrade your BIOS, they might be turned back off. (I am not sure about other toggles that could help, e.g., IOMMU. If you know, please help me improve this text!)
Talking about upgrading BIOS. I tend to follow the idiom "if it ain't broke, don't fix it". But my experience is that Xen needs BIOS fixes to run stably. 5 out of my 6 Xen systems were unstable under Xen first, but all became stable with a new, shining BIOS. They all had run stably under FreeBSD and/or NetBSD before I installed Xen and thus before I upgraded the BIOS.
We start with a NetBSD install. Get install media. I recommend using a late
any of the links, then grab an USB image
images/NetBSD-7.0_STABLE-amd64-install.img.gz. Uncompress it, and put it on a
USB stick using plain old
dd. Then boot the installation program from this stick.
I recommend these snapshots as earlier NetBSD kernels have a serious bug in Xen scheduling which will make DomU's useless as soon as CPU usage in Dom0 goes up.
I do not intend this as a NetBSD install guide, I just point out some aspects that are relevant for getting Xen to work.
Note that the NetBSD installer first asks for all partition sizes, then in a later screen lets you set parameters for the partitions.
If you intend to boot with grub, change the partition type of / from the default FFSv2 to FFSv1. The other partitions can be either type, but the default FFSv2 is generally better.
Choose to install pkgsrc when asked. If you forget to do this, look at the NetBSD pkgsrc docs how to post-install it manually.
Finish the installation, reboot into your new system.
Install needed packages:
(cd /usr/pkgsrc/sysutils/xenkernel45; make install clean clean-depends) (cd /usr/pkgsrc/sysutils/xentools45; make install clean clean-depends)
(There is now also xenkernel46/xentools46. I haven't yet tried those, my using of the '45 variants is not a recommendation against '46.)
Download the NetBSD DOM0 kernel. Then do:
gunzip netbsd-XEN3_DOM0.gz mv netbsd-XEN3_DOM0 /(Don't get confused by the references to XEN3 here; this kernel works with Xen 4.2.)
Copy the Xen kernel in place (yes, the gz file) and Xen startup files where the NetBSD boot code finds them:
cp -p /usr/pkg/xen45-kernel/xen.gz / cp -p /usr/pkg/share/examples/rc.d/xen* /etc/rc.d
Create needed Xen-related device files:
(cd /dev; ./MAKEDEV xen)(This creates devices for just 4 guests; use
for i in `seq 4 20`; do (cd /dev; ./MAKEDEV vnd$i); doneto make a few more.)
/boot.cfg. Here is the file I use:
menu=Boot normally:rndseed /var/db/entropy-file;boot netbsd menu=Boot single user:rndseed /var/db/entropy-file;boot netbsd -s menu=Disable ACPI:rndseed /var/db/entropy-file;boot netbsd -2 menu=Disable ACPI and SMP:rndseed /var/db/entropy-file;boot netbsd -12 menu=Boot Xen 1:load /netbsd-XEN3_DOM0 console=pc; multiboot /xen.gz dom0_mem=1024M menu=Drop to boot prompt:prompt default=5 timeout=5 clear=1(Again, don't get confused by the references to XEN3 here; this kernel works with Xen 4.x.)
A reboot here should work, except for networking of emulated systems.
The right approach depends on whether the emulated systems will have private (such as 10/8, 192.168/16) or public IP addresses. Or more specifically, if the "upstream" interface of the host is not in the same private/public category as the emulated systems interfaces.
If the upstream host interfaces is public and the emulated systems interfaces are private, then do:
bridge0. Put these lines in
auto_ifconfig=NO net_interfaces="`/sbin/ifconfig -l` tap0 bridge0"
tap0interface and assign it e.g., 192.168.0.1. Put this in
ifconfig_tap0="192.168.0.1 netmask 255.255.0.0"
/etc/ifconfig.bridge0with this contents:
create !brconfig $int add tap0 up
ext_if=XXX ext_addr=X.Y.Z.T nat on $ext_if from 192.168.0.0/16 to any -> $ext_addrin
XXXwith the host Ethernet interface of choice. Replace
X.Y.Z.Twith its IP address.
/etc/ifconfig.bridge0with this contents:
create !brconfig $int add XXX upReplace
XXXwith the host Ethernet interface of choice.
Done! A reboot command should now bring up the new Xen+NetBSD system with
functional networking, including network bridging for guests. If all is
xm li should list the guests. It should look something like
Name ID Mem VCPUs State time(s)
Time for some guests! It is beyond this text to describe how to do that. Important note: Do NOT use qemu-img-xen for making guest images, or else dd with the seek options, or any other way of creating sparse files. Sparse files and vnd are incompatible in NetBSD and using sparse files will cause the system to be quite instable. You might not see this problem with small guest images, but as images get larger, you WILL be out of luck.