Getting started with Xen virtualisation on Ubuntu 11.10

After a long time and lots of problems I decided to abandon openSuSE 11.4 and its xen implementation in favour of the PVOPS kernel and a different distribution.

It’s been difficult to choose the correct one for me, for now I’m working with Ubuntu 11.10. One reason is that it’s said to be user friendly, and highly customisable. It comes with all the right ingredients for running different hypervisors, including my favourite: xen.

Important update! See “Security” below.

Background on Xen

For those who don’t know the story behind xen, here’s a short summary (errors and omissions are not intentional!)

Xen allows para-virtualisation of operating systems. Unlike with hardware virtualisation you know from the likes of Virtual Box and VMware products, a para-virtualised operating system must be aware of the fact it is virtualised. This requires access to the source code, a reason why you can’t run para-virtualised Windows on Linux Xen. Microsoft has their own imitation of xen, called hyper-V but it lacks in usability and functionality.

Xen boots into a privileged hypervisor, called dom0 with exclusive access to the hardware. Guests (also known as virtual machines or domUs) interact with the hardware through the hypervisor.

After a while, xen was bought by Citrix. Xen sources initially weren’t part of the official kernel and had to be patched into it. This meant a kernel had to be compiled as well, something not everyone is comfortable doing. For one reason or antoher, Red Hat stopped doing so with Fedora 9, and Red Hat Linux 5.x is the last of their enterprise distributions to include dom0 support. Red Hat 5 is based on kernel 2.6.18.x and further back ports from upstream development. Kernel 2.6.18 is known as the “xenified kernel”, because of said patches.

At that stage it looked as if Xen was dead (at least to me). Processors were increasingly capable to perform bottlenecking operations in silicone rather than software speeding ESX up by lots. The benefits of para-virtualisation were less visible.

OpenSuSE is the only distribution carrying the xen patches forward to my knowledge; however their 11.4 distribution had lots of usability problems causing me to abandon it. OpenSuSE 12.1 seems to continue with the forward-carrying of patches.

However in years of hard work the xen patches were integrated into the mainline kernel, an interesting background story is provided by Wim Coekarts here: http://blogs.oracle.com/wim/entry/linux_mainline_contains_all_the.  The important change in kernel development was the move away from xenified kernels (which are separate from “the” kernel) to paravirtualised ops, or pvops for short. The pvops kernel contains the entire necessary infrastructure to either be a xen kernel, or “normal” desktop kernel. Fedora 16, Ubuntu and others use the PVOPS kernel, which requires xen 4.0.

With kernels 2.6.39+ for the first time it is possible to provide a dom0 and domU without the kernel patch.

My Ubuntu experience

I installed a minimal version of Ubuntu 11.10 server on my environment. To benefit from Xen virtualisation, the following packages plus their dependencies were required:

• xen-hypervisor-4.1-amd64
• xen-utils-4.1
• xenwatch
• xen-tools
• xen-utils-common
• xenstore-utils
• virtinst
• virt-viewer
• virt-manager

This creates a new menu for grub2 to boot into. The use of grub2 came as a little shock to me after I mastered lilo and “legacy” grub. To get on friendlier terms with it I installed startupmanager, and referred to help.ubuntu.com/community/Grub2, which is a great resource. A propos “install”. Being a RPM person, the use of apt-cache, dpkg and apt-get required a little getting used to. Luckily there are many debian/Ubuntu tutorials out there, including cheat sheets for these tools.

Using startupmanager I changed the default boot menu entry to my xen kernel and rebooted. Due to a bug in the ACPI implementation on the server I had to change /etc/default/grub file to acpi=ht.

Also, the /etc/xen/xend-config.sxp file needs to be amended to include (xend-unix-server yes)

Time for a reboot! After you got the console back, check that xen and virsh:

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 23149     8     r-----     16.6
# virsh version
Compiled against library: libvir 0.9.2
Using library: libvir 0.9.2
Using API: Xen 3.0.1
Running hypervisor: Xen 4.1.0

Networking

After the reboot I was surprised to see a “virbr0” interface in the output of ifconfig. I checked the /etc/network/interfaces file where Ubuntu stores its network configuration but couldn’t find any. The libvirt wiki (http://wiki.libvirt.org/page/Networking) describes the virbr0 interface as the “default” network created by libvirt at boot time, using NAT. The configuration file can be found in /usr/share/libvirt/networks/default.xml and it’s used for quick and dirty live CD or other boot media that aren’t used for production. The documentation states quite clearly that you shouldn’t manually add any interfaces to the bridge. The benefit of using the default network is the ease of use: a DHCP server distributes an IP address, and NAT works out of the box including the correct setting of firewall rules. However it’s not what I want.

What I wanted though was a setup similar to what I had before: host only networks, using br0 for the “public” network, br1 and br2 for RAC interconnect or iSCSI related storage. The tools to achieve this are in package bridge-utils. To set up a bridge, edit /etc/network/interfaces and add these directives:

auto br0
iface br0 inet static
pre-up brctl addbr br0
post-down brctl delbr br0
address 192.168.99.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

auto br1
iface br1 inet static
pre-up brctl addbr br1
post-down brctl delbr br1
address 192.168.100.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

auto br2
iface br2 inet static
pre-up brctl addbr br2
post-down brctl delbr br2
address 192.168.101.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

You could even bridge eth0 to it but that is not intended. My domUs are nicely hidden from the outside world.

More loop devices

Nearly there! I actually was in the process of adding shared storage to my server when I noticed I only had 8 loop devices available to me. Normally not a big deal – unload the “loop” module, and load it again with a different value for max_loop. However, the loop module is statically compiled into the kernel.

After a lot of fiddling and the helpful comment (see below) I managed to solve the problem by amending /etc/default/grub. It contains a default line for kernel arguments. Mine has the following content:

GRUB_CMDLINE_LINUX_DEFAULT=”max_loop=256 console=tty0 acpi=off”

I wouldn’t set acpi off if I didn’t have a problem with the server failing to reboot due to a fauly ACPI implementation. I also can’t experiment with different NUMA settings which is a shame.

Security

One thing I have forgotten to mention is the security aspect-my server didn’t have a host based firewall started right away. After trying to understand “SuSEFirewall” and IPTables I was very pleasantly surprised to learn about ufw. A few commands secure your environment, have a look at these 2 great resources for ubuntu 11.10:

• https://help.ubuntu.com/11.10/serverguide/C/firewall.html#firewall-ufw
• https://wiki.ubuntu.com/UncomplicatedFirewall

Summmary

I looks promising! Just need to work out more loop devices and I’ll be good to start installing the first operating system on the new system.