Category Archives: ESX

Oracle Linux support in ESXi

For quite some time now I am using ESXi 5 update 1 for my lab server and I’m very happy with it. In my lab environment I am not too picky what to run and do not worry about support too much. It’s not production!

One area of concern has been the support for Oracle’s own kernel: UEK or Unbreakable Enterprise Kernel. UEK comes in two editions, one based on 2.6.32, just like Red Hat’s kernel for Red Hat 6. The difference is that you can get UEK/1 ( for Oracle Linux 5.x as well instead of 2.6.18xxx which is otherwise the default.

Oracle’s second iteration of kernel UEK is unsurprisingly named UEK2 and it’s initially based on 3.x but keeps the name to 2.6.39.x for compatibility reasons. UEK2 has some really nice features taken from the Upstream kernel and it is also supported for the Oracle database.

Until not too long ago UEK was not supported by VMware ESXi, but this has changed without me taking notice at first. Thanks to a tweet by @dba_emc2 (Allan Robertson) I learned more about the change in the support policy. One interesting blog post from VMware is found here:

This post only mentions UEK, but does not clearly state whether UEK or UEK2 or both will be supported. The VMware Compatibility Guide has more information at

  • In the search, enter “unbreakable” to be directed the the relevant certification information
  • It turns out it is (at the time of this writing) UEK 2 actually which is great news for me!
  • Supported versions of ESXi are 5 u1, 5u2 and 5.1 at the time of writing
  • Support date is listed as 09/2012
  • There are even specific installation instruction but they don’t go over and above what you would normally do
  • What’s very interesting is that the paravirtualised drivers for SCSI (vSCSI) and VMXNet3 are supported too
  • You can also add virtual CPUs and memory while the VM is up (with the proper VM settings, I think hot-adding these is deactivated by default)

Enjoy! I will try to install Oracle Linux 6.3 – which is the first to my knowledge that boots UEK2 by default – next and install the VMware tools. Let’s see how that goes.


Why ESXi 5 has become my new standard hypervisor

OK so I have to admit that I was very sceptical at first about any non-paravirtualised hypervisors. Mainly because my knowledge about virtualisation seems to have become a little dated. I thought that anything that’s para-virtualised will clearly outrun anything else. Based on my experience with my older hardware this was actually true.

However, a couple of months ago I bought a new lab server, and because VMware kindly provided the licenses for ESXi 5 through the guru licensing scheme I gave it a try. I had a few initial problems with the fact that ESXi 5 doesn’t have the same shell as its predecessor, and it also uses the GPT format which didn’t work well with my Oracle Linux 6 installation using grub 0.97, which cannot deal with the MBR on a GPT disk.

However the nice thing about ESXi is the documentation which is very accessible to the newbie and well written. And then you have hundreds of blog posts for almost all problems you might encounter. I wanted to dual boot my system, and to make that possible I installed a new hard disk for Oracle Linux 6.2 (now 6.3) as the first disk in the machine. The BIOS setting is set AHCI to enable native command queuing, and my disk 1 is also mapped to disk 1 in the BIOS. I then installed Oracle Linux 6.2 on that particular disk, with GRUB in the MBR.

ESXi disks were added later, four in total for my various data stores. I then used a desktop virtualisation product to install ESXi 5 on a 16 GB USB key, off which I now boot when the serves starts. Simply selecting the BBS and then USB: off you go.

The main reason though for ESXi on my host is its stability. I am capable of running a lot of VMs with it and so far had no crashes whatsoever. I have also been able to virtualise a competing virtualisation platform on the same kit. The management interface in form of the vSphere client is superb and intuitive, and even VMware Workstation 8 has an interface to ESXi so you can manage local and remote VMs in an identical way. To be fair, the vSphere client is still needed though for advanced administration. All in all very neat indeed, at least for me.

The new lab server has arrived

Update: the Oracle Linux memory problem is solved:

As part of an on-going research project I have invested a little bit of money towards a new lab server. Sadly my Phenom II X6 with 16GB RAM was not on the HCL for ESXi 5 which I required. Before I knew that ESXi can itself be virtualised I looked at a number of options.

After much consideration and calculations I decided to buy a machine. I have already rented a Core i7-2600 with 32GB of RAM from Hetzner (which is superb value for money for their EX4-check it out!) but needed something more flexible in terms of OS installation and virtualisation. Although the team at Hetzner is most helpful and skilled, installing ESXi or Oracle VM 3.1.x would have been quite a task. Especially hardening the setup!

For some time I thought that it is no longer possible to virtualise certain Oracle setups on a ultra-mobile laptop. I managed to run a 2 node 10.2 cluster with ASM on a Think Pad T61 at the time, but when 11.2 (especially came out memory requirements pushed the laptops to their limit. Sure there are laptops that can be equipped with > 16GB RAM, but they can hardly be carried around. I needed a good compromise between battery endurance and weight, and came to the Think Pad X220 with a Core i5 2540M. This is a great laptop with 8 GB RAM and 320GB spinning disk. Thinking about it now I should have bought a SSD instead, but that can always be fitted later.

But I digress-back to the lab machine. The choices were either a consumer board, or a server/workstation setup. The consumer boards were quickly ruled out, as they are single socket only and I wanted a bit more.

The server boards for x86-64 can be separated in AMD and Intel. With AMD going down rapidly since the new processor architecture couldn’t compete with the latest Intel chips AMD has to stay in the market by offering a price advantage. The current 6200 series Interlagos chips come with 8 up to 16 cores per socket. Although I should add the “cores” in the 6200 processor are neither really cores nor threads, but that’s a different story, better told by wikipedia. The alternative is the Sandy Bridge E5-2600 series processor, better known as the processor to die for (not literally). This is currently some of the fastest you can get, and I really would have liked one. However the price tag associated with one of them is quite high-rightly so! Also the E5-2600 Xeon is made for dual socket workstations, the new 4600 series for quad-socket hasn’t been released yet at the time I made a decision to buy. And even then I wouldn’t have been able to afford one anyway.

As a regular reader of c’t, a very good German IT magazine I noticed Delta Computers in Reinbeck near Hamburg. They sell HPC kit to academic institutions but also offer hardware to everyone else. I got in touch (the website might need an overhaul) and quickly configured my system:

  • AMD 6238 Interlagos with 2 sockets @ 12 “cores” each
  • 32 GB RAM (the upgrade to 64 GB would only have been an extra 200 GBP)
  • No internal disks, I have plenty of these
  • Supermicro H8DG6 board with G34 chipset

So what’s nice about it? The price to start with, but also the IPMI 2.0 interface, the onboard SAS and SATA interface (no SATA 6g though). The case has a disk cage into which I can attach 8 SATA disks in caddies which requires no access to the mainboard and fumbline with screws.

I should also mention a little problem though: there is noise. The board has 5 fans attached I could see: 3 take the air in from the front and channel it across the CPU, the North Bridge and the memory. Two small fans suck the air out of the case and blow it out. They are small in diameter, and I had to find the BIOS setting to reduce their RPM. The familiar airplane-taking-off noise greeted me when I first powered the server up, and I thought I couldn’t possibly use it while anyone else is in the house. The case weights 35kg, and is a 4 U workstation. There is plenty of room for PCIe v2 x16 cards (3 I think) and the board takes up to 256GB of memory.

The IPMIView software from Supermicro allows me to control the server over a KVM-over-IP. I can even mount ISO images over the network which will replace the DVD ROM. It is possible to administer the box without ever plugging a keyboard, mouse, monitor or physical boot device into it.

ESXi 5 installs beautifully and is fully Interlagos-compatible. Some guides on the Internet suggest a little tweaking of the advanced settings for ESXi to enable the deep C-states, allowing unused cores to sleep when they are not used.

On the downside I couldn’t make Oracle Linux 6.2 with kernel UEK detect all the memory. In fact, of my 32GB only 3 remained. This seems to be a regression introduced in 2.6.28 (or so) which is supposedly fixed in Red Hat 6.2. There is also a list available on Supermicro’s site listing which operating systems are optimised for the Interlagos chips. There is a possibility of excessive cache invalidations in systems not aware of the 6200 architecture. ESXi 5 is aware of it, so I stick with it for the time being.

After almost a month I have to say I’m very happy with the server. It’s noisy, ok, but having headphones with good music makes it bearable. The performance is great, I have yet to try and see how well it copes with my 4 node RAC cluster and OEM 12.1 Cloud Control.

NIC confusion when cloning virtual machines in ESX5i

A few twitter posts have already indicated that I have a new machine in my study-a Supermicro server with the H8DGi6-F motherboard, 2 12-core AMD 6238 processors and 32 GB RAM. And since I have been given a vSphere 5 Enterprise Plus license, what better to test than this combination! I have to say the installation of ESX5i is very straight forward: get the ISO image, burn it, install it. Done-and ESX5i has support for the AMD 6200 processors built in, which suffer from L1 cache invalidations in certain situations.

I have to say that ESX5i is a data center product: I haven’t been able to attach a USB hard disk to it in order to copy my ISO images. Apparently there is a way for ESX 5i (and 4i) to mount vfat formatted USB disks to automatically mount the devices as “NO NAME” but I haven’t been able to get this to work.

Also, the networking is a bit different from xen which I have used extensively before. But it’s all logical and not an issue once you understand it. My first virtual machine is an Oracle Linux 6.2 infrastructure host. Cloning a VM is actually quite simple-in the vSphere client you browse the datastore, create a new folder and copy source into destination folder. Then you right-click the VMX file and “add to inventory”: done!

The first time you start the cloned VM however there is a problem: the MAC addresses for the machine will change after you tell vSphere that you “copied” the virtual machine. This initialises the VMX file with a new UUID and MAC addresses for the VMs. The same is true for a cloning operation in Virtual Box for example. New MAC address will prompt kuduzu to consider the NICs as new devices. In my case, the NICs were eth0 and eth1 on the source, and subsequently ended up being named eth2 and eth3 in the clone-and no eth0 and eth1. Whoops!

But that’s really, really simple to change. In you VM settings, note down the MAC address for each NIC you have, and then open the file /etc/udev/rules.d/70-persistent-net.rules. You may see something like this:

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:5d:d9:10", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:5d:d9:06", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:e2:8f:58", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:e2:8f:4e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

You can see where this is going… I simply copied and pasted the new MAC addresses into the old udev rule and removed the new ones, as shown here (remember to back-reference to the settings you just wrote down)

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:e2:8f:58", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:e2:8f:4e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Now all I needed to do was run system-config-network to update /etc/sysconfig/network-scripts/ifcfg-eth{0,1} with the new MAC addresses and start the network. Result:

[root@ol62infra ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:E2:8F:4E
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::20c:29ff:fee2:8f4e/64 Scope:Link
          RX packets:54 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7818 (7.6 KiB)  TX bytes:1669 (1.6 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0C:29:E2:8F:58
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::20c:29ff:fee2:8f58/64 Scope:Link
          RX packets:257 errors:0 dropped:0 overruns:0 frame:0
          TX packets:193 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:26218 (25.6 KiB)  TX bytes:25039 (24.4 KiB)

lo        Link encap:Local Loopback
          inet addr:  Mask:
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

See, cloning isn’t that hard after all!