Martins Blog

Trying to explain complex things in simple terms

Archive for the ‘Xen’ Category

RAC in KVM is possible without iSCSI

Posted by Martin Bach on September 3, 2013

(This post is for Jerry. He will know when he reads it)

I have been a great supporter of many flavours of virtualisation and my earliest experience with Xen goes back to Oracle VM 2 which was based on RHEL 4 and an early version of Xen. Why am I saying this? Because Xen is (was?) simple and elegant. Especially for building RAC systems: paravirtualised Linux was all you needed, and a dual-core machine: Xen is very lightweight even though recent achievements in processor architecture (nested page tables, single root IO virtualisation, others) make it more desirable to use hardware virtualisation with paravirtualised drivers. This is what this post is about!

Shared storage in Xen

As you know you need shared block devices for RAC for voting disks, OCR, data files, redo logs, the lot. In Xen that’s as straight forward as it gets:

disk = [

The “w!” means the block device is shared. Simple. Make the file available to the second RAC node and you are done.

In 2009 (yikes!) I ask the question if you could have RAC on KVM with the same technique. Who would have known then that Karl would be a colleague on day? Fast forward a few years and I have implemented such a solution. And it works in the lab. Read the rest of this entry »

Posted in KVM, Linux, Xen | 3 Comments »

Availability Infrastructure & Management SIG March 14th 2012

Posted by Martin Bach on March 3, 2012

I am proud to be able to speak at the first instalment of the Availability, infrastructure and management SIG on March 14th in  the London City office.

The event is announced on the UKOUG website here:

Unfortunately I will be between you and lunch! I hope that works out, and I don’t overrun.

I am going to demonstrate my (little) knowledge of Oracle Enterprise Manager 12.1. I looked at it for one of my customers and came to like it. As it is very different from the previous versions of the product, a more closely focused session seems appropriate. An Internet connection permitting, I am going to demonstrate navigation through the new interface, self update, target discovery and if time permits, I will patch a single instance HA environment (also known as Oracle Restart).

If all demos work this could be quite an entertaining sessions, questions are welcome!

Posted in Grid Control, Linux, Public Appearances, Xen | 4 Comments »

Did you know the cluvfy healthcheck?

Posted by Martin Bach on January 28, 2012

While I was performing a three day seminar recently in Switzerland I came across this new option in cluvfy.

Normally you’d run cluvfy in preparation of the installation of Grid Infrastructure or a set of RAC binaries to ensure everything is ready for the next step in the RAC install process. Beginning with, there is another option that’s been sneaked in without too much advertisement: the healthcheck.

Part of the “comp” checks, it takes the following options:

cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name] [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]

Read the rest of this entry »

Posted in 11g Release 2, Linux, Xen | Tagged: , | 5 Comments »

Getting started with Xen virtualisation on Ubuntu 11.10

Posted by Martin Bach on November 30, 2011

After a long time and lots of problems I decided to abandon openSuSE 11.4 and its xen implementation in favour of the PVOPS kernel and a different distribution.

It’s been difficult to choose the correct one for me, for now I’m working with Ubuntu 11.10. One reason is that it’s said to be user friendly, and highly customisable. It comes with all the right ingredients for running different hypervisors, including my favourite: xen.

Important update! See “Security” below.

Read the rest of this entry »

Posted in Linux, Xen | 7 Comments »

Move the EM12c repository database

Posted by Martin Bach on October 17, 2011

I have made a little mistake creating a RAC database for the OEM 12c repository-I now need a little more lightweight solution, especially since I’m going to do some fancy failover testing with this cluster soon! An single instance database without ASM, that’s what I’ll have!

Now how to move the repository database? I have to admit I haven’t done this before, so the plan I came up with is:

  1. Shut down the OMS
  2. Create a backup of the database
  3. Transfer the backup to the destination host
  4. Restore database
  5. Update OEM configuration
  6. Start OMS

Read the rest of this entry »

Posted in Grid Control, Linux, Xen | 4 Comments »

The tale of restoring the OCR and voting files on Linux for RAC

Posted by Martin Bach on October 13, 2011

As part of a server move from one data centre to another I enjoyed working in the depths of Clusterware. This one has been a rather simple case though: the public IP addresses were the only part of the package to change: simple. One caveat though was the recreation of the OCR disk group I am using for the OCR and 3 copies of the voting file. I decided to reply on the backups I took before the server move.

Once the kit has been rewired in the new data centre, it was time to get active. The /etc/multipath.conf file had to be touched to add the new LUNs for my +OCR disk group. I have described the processes in a number of articles, for example here:

A few facts before we start:

  • Oracle Enterprise Linux 5.5 64bit
  • device-mapper-multipath-0.4.7
  • Grid Infrastructure (actually it is Oracle Database SAP Bundle Patch
  • ASMLib

I have already described how to restore the OCR and voting files in in “Pro Oracle Database RAC 11g on Linux”, but since then the procedure has changed slightly I thought I’d add this here. The emphasis is on “slightly”. Read the rest of this entry »

Posted in 11g Release 2, Automatic Storage Management, RAC, Xen | 2 Comments »

Installing Oracle Enterprise Manager 12c on OL 5.7

Posted by Martin Bach on October 7, 2011

I have been closely involved in the upgrade discussion of my current customer’s Enterprise Managers setup from an engineering point of view. The client uses OEM extensively for monitoring, alerts generated by it are automatically forwarded to an IBM product called Netcool.

Now some of the management servers are still on in certain regions, and for a private cloud project I was involved in an 11.1 system was needed.The big question was: wait for 12.1 or upgrade to 11.1?

So to cut a long story short I have been very keen to get to the OEM 12c beta programme, but unfortunately wasn’t able to make it. Also, I wasn’t at Open World this year which means I didn’t get to see any of the demos. You can imagine I was quite curious to get my hands on it, and when it has been released a few days ago I downloaded it to my lab machine. I created a new domU for the database- plus latest PSU and another one for the management server. I assigned 2 CPUs each, the database server got 2G of memory while the OMS received 8.Don’t take this as a recommendation though, it’s only for lab use! I wouldn’t use less than 24G of memory for a production management server, and it would obviously follow the MAA recommendations and be installed behind an enterprise grade load balancer etc. Needless to say I’d use RAC+Data Guard for the repository database.

Read the rest of this entry »

Posted in Grid Control, Linux, Xen | Tagged: | 4 Comments »

Adding another node for RAC on Oracle Linux 6.1 with kernel-UEK

Posted by Martin Bach on October 6, 2011

As I have hinted at during my last post about installing Oracle on Oracle Linux 6.1 with Kernel UEK, I have planned another article about adding a node to a cluster.

I deliberately started the installation of my RAC system with only one node to allow my moderately spec’d hardware to deal with a second cluster node. In previous versions of Oracle there was a problem with node additions: the $GRID_HOME/oui/bin/ script did pre-requisite checks that used to fail when you had used ASMLib. Unfortuntely, due to my setup I couldn’t test if that was solved (I didn’t use ASMLib).


As with many cluster operations on non-Exadata you should use the cluvfy tool to ensure that the system you want to add to the cluster meets the requirements. Here’s an example session for the cluvfy output. Since I am about to add a node, the stage has to be “-pre nodeadd”. rac11203node1 is the active cluster node, and rac11203node2 the one I want to add. Note that you run the command from (any) existing node, specifying the nodes to be added with the “-n” parameter. For convenience I have added the “-fixup” option to generate fixup scripts if needed. Also note that this is a lab environment, real production environments would use dm-multipath for storage and a bonded pair of NICs for the public network. Since you no longer need to bond your private NICs, Oracle does that for you now.

Read the rest of this entry »

Posted in 11g Release 2, Linux, Xen | Tagged: , | 2 Comments »

Installing Grid Infrastructure on Oracle Linux 6.1 with kernel UEK

Posted by Martin Bach on September 24, 2011

Installing Grid Infrastructure on Oracle Linux 6.1

Yesterday was the big day, or the day Oracle release for Linux x86 and x86-64. Time to download and experiment! The following assumes you have already configured RAC 11g Release 2 before, it’s not a step by step guide how to do this. I expect those to shoot out of the grass like mushrooms in the next few days, especially since the weekend allows people to do the same I did!

The Operating System

I have prepared a xen domU for, using the latest Oracle Linux 6.1 build I could find. In summary, I am using the following settings:

  • Oracle Linux 6.1 64-bit
  • Oracle Linux Server-uek (2.6.32-100.34.1.el6uek.x86_64)
  • Initially installed to use the “database server” package group
  • 3 NICs – 2 for the HAIP resource and the private interconnect with IP addresses in the ranges of and The public NIC is on
    • Node 1 uses 192.168.(99|100|101).129 for eth0, eth1 and eth2. The VIP uses
    • Node 1 uses 192.168.(99|100|101).131 for eth0, eth1 and eth2. The VIP uses
    • The SCAN is on 192.168.99.(133|134|135)
    • All naming resolution is done via my dom0 bind9 server
  • I am using a 8GB virtual disk for the operating system, and a 20G LUN for the oracle Grid and RDBMS homes. The 20G are subdivided into 2 LVMs of 10G each mounted to /u01/app/oracle and /u01/crs/ Note you now seem to need 7.5 G for GRID_HOME
  • All software is owned by Oracle
  • Shared storage is provided by the xen blktap driver
    • 3 x 1G LUNs for +OCR containing OCR and voting disks
    • 1 x 10G for +DATA
    • 1 x 10G for +RECO

Read the rest of this entry »

Posted in 11g Release 2, Linux, Xen | Tagged: | 4 Comments »

An introduction to collectl

Posted by Martin Bach on August 5, 2011

Some of you may have seen on twitter that I was working on understanding collectl. So why did I start with this? First of all, I was after a tool that records a lot of information on a Linux box. It can also play information back, but this is out of scope of this introduction.

In the past I have used nmon to do similar things, and still love it for what it does. Especially in conjunction with the nmon-analyzer, an Excel plug in it can create very impressive reports. How does collectl compare?

Getting collectl

Getting collectl is quite easy-get it from sourceforge:

The project website including very good documentation is available from sourceforge as well, but uses a slightly different URL:

I suggest you get the archive-independent RPM and install it on your system. This is all you need to get started! The impatient could type “collectl” at the command prompt now to get some information. Let’s have a look at the output:

$ collectl
waiting for 1 second sample...
#cpu sys inter  ctxsw KBRead  Reads KBWrit Writes   KBIn  PktIn  KBOut  PktOut
1   0  1163  10496    113     14     18      4      8     55      5      19
0   0  1046  10544      0      0      2      3    164    195     30      60
0   0  1279  10603    144      9    746    148     20     67     11      19
3   0  1168  10615    144      9    414     69     14     69      5      20
1   0  1121  10416    362     28    225     19     11     71      8      35

The “ouch” has been caused by my CTRL-c to stop the execution.

Collectl is organised to work by subsystems, the standard option is to print CPU, disk and network subsystem, aggregated.

Read the rest of this entry »

Posted in 11g Release 2, Linux, Performance, Perl, Xen | Tagged: , , , | 11 Comments »