Build your own RAC system part IV – extending Grid Infrastructure

UPDATE 221103: Oracle 11.2 is effectively out of support, this article is now archived and should not be referred to

After having experimented with my Clusterware setup in the lab, I thought it was time to put the promise of easy extensibility to the test and run to extend the cluster to two nodes. That used to be quite easy in 10.2 and 11.1: the fired up the GUI, you filled in the new public, private and virtual IP and off it went. A few script executions as root later you had an extended cluster. Well, you still needed to add the ASM and RDBMS homes but that was a piece of cake after the Clusterware extension. How easily could that be done with 11.2?

CLUVFY enhancements

Some of the enhanced options in cuvfy are really handy in this case. You could run it with “comp peer” and “post –hwos” to check for any problems beforehand. The ultimate weapon though is the “stage –pre nodeadd”. You execute the script on one of the existing nodes and fill in the new node name. The output of the command is trimmed down for readability

[oracle@rac11gr2node1 ~]$ cluvfy stage -pre  nodeadd -n rac11gr2node2  -fixup -fixupdir /tmp

Performing pre-checks for node addition

Checking node reachability...
Node reachability check passed from node "rac11gr2node1"

Check: Node connectivity for interface "eth0"
Node connectivity passed for interface "eth0"

Node connectivity check passed

Node connectivity passed for subnet "" with node(s) rac11gr2node1,rac11gr2node2
TCP connectivity check passed for subnet ""

Node connectivity passed for subnet "" with node(s) rac11gr2node1,rac11gr2node2
TCP connectivity check passed for subnet ""

Interfaces found on subnet "" that are likely candidates for VIP are:
rac11gr2node1 eth0: eth0: ...
rac11gr2node2 eth0:

Interfaces found on subnet "" that are likely candidates for a private interconnect are:
rac11gr2node1 eth1:
rac11gr2node2 eth1:

Node connectivity check passed
PRVF-5415 : Check to see if NTP daemon is running failed
Clock synchronization check using Network Time Protocol(NTP) failed

Fixup information has been generated for following node(s):
Please run the following script on each node as "root" user to execute the fixups:

Pre-check for node addition was unsuccessful on all the nodes.
[oracle@rac11gr2node1 ~]$

Whatever I tried, it always complained about the NTP daemon-I thought ctss would take care of cluster time synchronisation. It doesn’t affect the cluster so nothing to worry too much about.

The fixup option allows you to create a script to address fixable problems, such as missing/wrong kernel parameters. You execute the script on the node you’d like to add:

[root@rac11gr2node2 CVU_11.]# ./
Response file being used is :./fixup.response
Enable file being used is :./fixup.enable
Log file location: ./orarun.log
Setting Kernel Parameters...
fs.file-max = 327679
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.wmem_max = 262144
net.core.wmem_max = 1048576

That’s great! As with many of these features you wonder why they haven’t been implemented before!

The next step then involves invoking in $ORA_CRS_HOME/oui/bin. Another change – is now “headless”, i.e. no more GUI installation, even if you wanted. A silent installation was possible before, but for me that meant reading up on the OUI and OPatch guide for Linux and Windows. Furthermore, you have different options depending on whether you use GNS or not. Whilst the documentation claims all that was needed to extend the Grid Infrastructure with GNS was a quick “ –silent “CLUSTER_NEW_NODES={newNodePublicIP}” this didn’t work. My OUI session consistently complained about this:

SEVERE:Number of new nodes being added are not equal to number of new virtual nodes. Silent install cannot continue.

As with all OUI sessions, you’ll find a set of logs in $ORACLE_BASE/oraInventory/logs/

I already suspected that OUI wasn’t aware of GNS. I have raised it with Oracle-there wasn’t a OUI patch out for 11.2 yet when I looked for it. How could that have gone past testing? GNS is one of the most interesting features in 11.2, surely it must have been tested? Maybe not.

Passing the non-GNS syntax actually worked even in my case, and I successfully extended the cluster to two nodes. After the execution of even the oratab was setup correctly and defined +ASM2. I was pleasantly surprised.

The command that worked referenced CLUSTER_NEW_NAMES and CLUSTER_NEW_VIRTUAL_HOSTNAMES:

./ -silent "CLUSTER_NEW_NODES={rac11gr2node2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac11gr2node2-vip}"

Output from

Using username "oracle".
Authenticating with public key "martins key" from agent
Last login: Sun Oct 11 20:50:47 2009 from
[oracle@rac11gr2node2 ~]$ su -
[root@rac11gr2node2 ~]# /u01/app/oraInventory/
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac11gr2node2 ~]# /u01/crs/oracle/product/11.2.0/grid/ 2>&1 | tee /tmp/
Running Oracle 11g script...

The following environment variables are set as:
 ORACLE_HOME=  /u01/crs/oracle/product/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
 Copying dbhome to /usr/local/bin ...
 Copying oraenv to /usr/local/bin ...
 Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
CRS-2672: Attempting to start 'ora.evmd' on 'rac11gr2node2'
CRS-2676: Start of 'ora.evmd' on 'rac11gr2node2' succeeded
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.

rac11gr2node2     2009/10/11 20:57:50     /u01/crs/oracle/product/11.2.0/grid/cdata/rac11gr2node2/backup_20091011_205750.olr
Preparing packages for installation...
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 511 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[root@rac11gr2node2 ~]# uname -a
Linux rac11gr2node2 2.6.18-164.el5xen #1 SMP Thu Sep 3 02:41:56 EDT 2009 i686 i686 i386 GNU/Linux
[root@rac11gr2node2 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
[root@rac11gr2node2 ~]# grep -i name /proc/cpuinfo
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
[root@rac11gr2node2 ~]#

So with the solid foundation in place I’m now ready to install the RDBMS software now, the goal is to experiment with RAC One Node. I know it’s a slightly flawed test running this out of Oracle VM but I don’t have the hardware, especially the SAN to test a different setup. Or maybe I should experiment with OpenFiler and iSCSI…



5 thoughts on “Build your own RAC system part IV – extending Grid Infrastructure

  1. Steve

    RE: The following error:
    PRVF-5415 : Check to see if NTP daemon is running failed
    Clock synchronization check using Network Time Protocol(NTP) failed
    I was above to correct this on my local systems by changing the line in the /etc/sysconfig/ntpd file:
    OPTIONS=”-u ntp:ntp -p /var/run/”
    OPTIONS=”-x -u ntp:ntp -p /var/run/”
    Note the addition of the “-x”
    Thanks – Steve

  2. john

    Hi, I have an exam question, still unsure about the answer.
    Appreciate if you help

    Your production envrionment cluster is running Oracle Enterprise Linux and currently has four nodes.
    You are asked to plan for extending the cluster to six nodes. Which three methods are availavle to add the new nodes ?

    a-)silent clonening using crsctl clone cluster and ssh
    b-)a GUI interface from Enterprise Manager
    c-)with the Oracle Universal Installer using runInstaller -clone
    d-)silent cloning using perl -silent either with parameter in a file or in line

    1. Martin Bach Post author

      Hi John,

      thanks for passing by. The idea of the RAC certified expert exam is to encourage people to try things out in their own environments, which is the point at when you learn most. I would like to encourage you to build a one node RAC system, and extend it to multiple nodes using your preferred approach. There are plenty of resources out there, including this blog which guide you through the process. You will appreciate this encouragement to learn and understand RAC once you sit in the exam.


  3. john

    Hi Martin

    Thanks for your response.
    I have found good articles about adding a new node in rac environment.
    I know that I can add a node by and silent cloning perl (d)
    I am just wondering whether this can also be done using Oracle Universal Installer (c) or crsctl clone cluster (a)
    I know that there is no way to do it with Enterprise Manager

Comments are closed.