Employing the same method, I ran into problems with Oracle 11.2.0.x clients. A few facts to start with:

• Oracle 11.2.0.3 client 64bit
• Golden image created on Oracel Linux 5
• Destination: SuSE Enterprise 10 SP4

The problem described here is most likely applicable to other Oracle clients as well although I haven’t verified that.

The problem

After the clone of the client to the SuSE server I couldn’t start SQL*Plus. Fair enough, I hadn’t set the LD_LIBRARY_PATH. After that I still couldn’t launch sqlplus because of a segmention fault. So if the clone wants to play it difficult, then I can always try to relink it all. But to my great astonishment that didn’t solve the problem either! Still same error.

The relink operation writes information into a logfile, called $ORACLE_HOME/install/relink.log. Please note that this isn’t the RDBMS home, when referring to ORACLE_HOME in this article I specifically speak of the client home!

The log looked normal until I came to this line:

/u01/app/oracle/product/client/11.2.0.3/bin/genclntsh
genclntsh: genclntsh: Could not locate  /u01/app/oracle/product/client/11.2.0.3/network/admin/shrept.lst
make: *** [client_sharedlib] Error 1
Error in invoking target 'client_sharedlib' of makefile '/u01/app/oracle/product/client/11.2.0.3/rdbms/lib/ins_rdbms.mk'. See '/u01/app/oracle/product/client/11.2.0.3/install/relinkActions2012-01-24_09-46-10-AM.log' for details.

Well ok then. In fact, there was no file in $ORACLE_HOME/network/admin at all. Not on the clone, neither in the golden image. This looked like a bug to me and I opened a SR with Oracle. The suggestion (unsurprisingly) was to copy the file either from the 11.2.0.3 RDBMS installation media or alternatively copy the file from an existing 11.2.0.3 RDBMS home.

Trying the relink operation now proved that the file was indeed essential for a relink operation. SQL*Plus now works.

Normally you’d run cluvfy in preparation of the installation of Grid Infrastructure or a set of RAC binaries to ensure everything is ready for the next step in the RAC install process. Beginning with 11.2.0.3, there is another option that’s been sneaked in without too much advertisement: the healthcheck.

Part of the “comp” checks, it takes the following options:

cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name] [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]

The most extensive report is run without any options, as shown in the appendix (the output is too long to display at this stage of the post) You have the following options:

• collect: you can either specify to collect information about the cluster, or the database. Don’t specify the collect option and the health check will collect information about both. See also the next option if you want to collect database related information!
• db: database unique name to collect information for. Before you can use this option, you need to run the script cvusys.sql in $GRID_HOME/cv/admin. It will create a user the healthcheck tool will connect against to find recommendations. If omitted, all cluster databases registered in the OCR will be probed.
• bestpractice, mandatory, deviations are self explanatory. As always, don’t take the “best practice” literally!
• html: Creates a HTML report in savedir, see below. If not used, a text based output is created
• save, savedir: save the report in savedir

Now this seems to be a nice way to produce a report at the end of the RAC installation which shows that the system is correctly set up.

Appendix

Here is a text version of a full report I generated on my 2 node cluster, comprised of rac11203node1 and rac11203node2. I didn’t have a database ready at the time hence the limitation to the cluster health check.

******************************************************************************************
Summary of environment
******************************************************************************************

Date (mm/dd/yyyy)    :  01/28/2012
Time (hh:mm:ss)      :  21:57:03
Cluster name         :  rac11203
Clusterware version  :  11.2.0.3.0
Grid home            :  /u01/crs/11.2.0.3
Grid User            :  oracle
Operating system     :  Linux2.6.32-100.34.1.el6uek.x86_64

******************************************************************************************
System requirements
******************************************************************************************

Verification Check          :  Network Time Protocol (NTP)
Verification Description    :  This task verifies cluster time synchronization on
clusters that use Network Time Protocol (NTP).
Verification Result         :  FAILED
Cluster-wide error details  :  Error -  If you plan to use CTSS for time
synchronization then NTP configuration must be
uninstalled on all nodes of the cluster.

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       FAILED    not available                 not available
rac11203node1       FAILED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Swap Size
Verification Description  :  This is a prerequisite condition to test whether
sufficient total swap space is available on the system.
Verification Result       :  WARNING
Verification Summary      :  Swap configuration did not meet the recommended value of
null on rac11203node2,rac11203node1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       FAILED    3.9127GB (4102768.0KB)        1023.9922MB (1048568.0KB)
rac11203node1       FAILED    3.9127GB (4102768.0KB)        1023.9922MB (1048568.0KB)

__________________________________________________________________________________________

Verification Check        :  Time zone consistency
Verification Description  :  This task checks for the consistency of time zones across
systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Task resolv.conf Integrity
Verification Description  :  This task checks consistency of file /etc/resolv.conf file
across nodes
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  User Mask
Verification Description  :  This is a prerequisite condition to make sure the user
file creation mask (umask) is "0022".
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    0022                          0022
rac11203node1       PASSED    0022                          0022

__________________________________________________________________________________________

Verification Check        :  User Not In Group: oracle
Verification Description  :  This is a prerequisite condition to make sure user
"oracle" is not part of "root" group.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Same core file name pattern
Verification Description  :  This task checks the consistency of core file name pattern
across systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Root user consistency
Verification Description  :  This test checks the consistency of the primary group of
the root user across the cluster nodes
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Current Group ID
Verification Description  :  This test verifies that the user is currently logged in to
the user's primary group.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Users With Same UID
Verification Description  :  This test checks that multiple users do not exist with
user id as "0".
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  OS Kernel Version
Verification Description  :  This is a prerequisite condition to test whether the
system kernel version is at least "2.6.32".
Verification Result       :  PASSED
Verification Summary      :  Kernel version meets recommendation

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    2.6.32                        2.6.32-100.34.1.el6uek.x86_6
4
rac11203node1       PASSED    2.6.32                        2.6.32-100.34.1.el6uek.x86_6
4

__________________________________________________________________________________________

Verification Check        :  Architecture
Verification Description  :  This is a prerequisite condition to test whether the
system has a certified architecture.
Verification Result       :  PASSED
Verification Summary      :  Architecture recommendation is met

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    x86_64                        x86_64
rac11203node1       PASSED    x86_64                        x86_64

__________________________________________________________________________________________

Verification Check        :  Soft Limit: maximum user processes
Verification Description  :  This is a prerequisite condition to test whether the soft
limit for "maximum user processes" is set to at least 2047.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    2047                          2047
rac11203node1       PASSED    2047                          2047

__________________________________________________________________________________________

Verification Check        :  Hard Limit: maximum user processes
Verification Description  :  This is a prerequisite condition to test whether the hard
limit for "maximum user processes" is set to at least
16384.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    16384                         16384
rac11203node1       PASSED    16384                         16384

__________________________________________________________________________________________

Verification Check        :  Soft Limit: maximum open file descriptors
Verification Description  :  This is a prerequisite condition to test whether the soft
limit for "maximum open file descriptors" is set to at
least 1024.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    1024                          1024
rac11203node1       PASSED    1024                          1024

__________________________________________________________________________________________

Verification Check        :  Hard Limit: maximum open file descriptors
Verification Description  :  This is a prerequisite condition to test whether the hard
limit for "maximum open file descriptors" is set to at
least 65536.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    65536                         65536
rac11203node1       PASSED    65536                         65536

__________________________________________________________________________________________

Verification Check        :  Run Level
Verification Description  :  This is a prerequisite condition to test whether the
system is running with proper run level.
Verification Result       :  PASSED
Verification Summary      :  Run level recommendation are met

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    3,5                           3
rac11203node1       PASSED    3,5                           3

__________________________________________________________________________________________

Verification Check        :  Group Membership: oinstall(Primary)
Verification Description  :  This is a prerequisite condition to test whether user
"oracle" has group "oinstall" as its primary group.
Verification Result       :  PASSED
Verification Summary      :  Group oinstall is the primary group of user oracle

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Group Existence: oinstall
Verification Description  :  This is a prerequisite condition to test whether group
"oinstall" exists on the system.
Verification Result       :  PASSED
Verification Summary      :  Group oinstall exists

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    true                          true
rac11203node1       PASSED    true                          true

__________________________________________________________________________________________

Verification Check        :  User Existence: oracle
Verification Description  :  This is a prerequisite condition to test whether user
"oracle" exists on the system.
Verification Result       :  PASSED
Verification Summary      :  User oracle exists

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    true                          true
rac11203node1       PASSED    true                          true

__________________________________________________________________________________________

Verification Check        :  Available Physical Memory
Verification Description  :  This is a prerequisite condition to test whether the
system has at least 50MB (51200.0KB) of available physical
memory.
Verification Result       :  PASSED
Verification Summary      :  Available memory meets or exceeds recommendation

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    50MB (51200.0KB)              3.029GB (3176136.0KB)
rac11203node1       PASSED    50MB (51200.0KB)              2.7401GB (2873240.0KB)

__________________________________________________________________________________________

Verification Check        :  Physical Memory
Verification Description  :  This is a prerequisite condition to test whether the
system has at least 1.5GB (1572864.0KB) of total physical
memory.
Verification Result       :  PASSED
Verification Summary      :  Physical memory meets or exceeds recommendation

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    1.5GB (1572864.0KB)           3.9127GB (4102768.0KB)
rac11203node1       PASSED    1.5GB (1572864.0KB)           3.9127GB (4102768.0KB)

__________________________________________________________________________________________

Verification Check        :  ASMLib installation and configuration verification.
Verification Description  :  This task checks the ASMLib installation and configuration
across the systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Multicast check
Verification Description  :  This task checks that network interfaces in subnet are
able to communicate over multicast IP address
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Node Connectivity
Verification Description  :  This is a prerequisite condition to test whether
connectivity exists amongst all the nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________


******************************************************************************************
System recommendations
******************************************************************************************

Verification Check        :  Reverse path filter setting
Verification Description  :  Checks if reverse path filter setting for all private
interconnect network interfaces is correct
Verification Result       :  MET
Verification Summary      :  Check for Reverse path filter setting passed
Additional Details        :  Reverse path filter parameter "rp_filter" must be set to
the value of 0 or 2 for all the private interconnect
network interfaces, this will disable or relax the
filtering and allow Clusterware to function correctly

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    0|2                           0|2
rac11203node1       PASSED    0|2                           0|2

__________________________________________________________________________________________

Verification Check        :  availability of port 8888
Verification Description  :  availability of port 8888
Verification Result       :  MET
Verification Summary      :  Check for availability of port 8888 passed

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    true                          true
rac11203node1       PASSED    true                          true

__________________________________________________________________________________________

Verification Check        :  Hardware Clock synchronization at shutdown
Verification Description  :  Checks whether Hardware Clock is synchronized with the
system clock during system shutdown
Verification Result       :  MET
Verification Summary      :  Check for Hardware Clock synchronization at shutdown passed

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    HWCLOCKSYNC=1                 HWCLOCKSYNC=1
rac11203node1       PASSED    HWCLOCKSYNC=1                 HWCLOCKSYNC=1

__________________________________________________________________________________________


******************************************************************************************
Clusterware requirements
******************************************************************************************

Verification Check          :  Clock Synchronization
Verification Description    :  This test checks the Oracle Cluster Time Synchronization
Services across the cluster nodes.
Verification Result         :  FAILED
Cluster-wide error details  :  Error -  Look at the accompanying error messages for the
nodes on which the check failed and fix the problem. If
you plan to use CTSS for time synchronization then NTP
configuration should be uninstalled on all nodes of the
cluster. Refer to "Preparing Your Cluster" of "Oracle
Database 2 Day+ Real Application Clusters Guide".

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       FAILED    not available                 not available
rac11203node1       FAILED    not available                 not available

__________________________________________________________________________________________

Verification Check          :  Node Connectivity
Verification Description    :  This is a prerequisite condition to test whether
connectivity exists amongst all the nodes. The
connectivity is being tested for the subnets
"192.168.99.0,192.168.100.0,192.168.101.0"
Verification Result         :  FAILED
Cluster-wide error details  :  Error -  Verify the interface configurations for the
network interfaces identified on the nodes indicated
using utilities like ipconfig or ping.

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node1 :     FAILED    not available                 not available
192.168.99.27
rac11203node2       FAILED    not available                 not available
rac11203node1       FAILED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  VIP Subnet configuration check
Verification Description  :  This task checks that all VIP subnetworks match each other
and at least one public network interface of the cluster
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  User Not In Group: oracle
Verification Description  :  This is a prerequisite condition to make sure user
"oracle" is not part of "root" group.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  ACFS Integrity
Verification Description  :  This test checks the integrity of Oracle ASM Cluster File
System across the cluster nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  OLR Integrity
Verification Description  :  This test checks the integrity of OLR on the local node.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Single Client Access Name (SCAN)
Verification Description  :  This test verifies the Single Client Access Name
configuration.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Node Application Existence
Verification Description  :  This test checks the existence of Node Applications on the
system.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  CRS Integrity
Verification Description  :  This test checks the integrity of Oracle Clusterware stack
across the cluster nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  OCR Integrity
Verification Description  :  This test checks the integrity of OCR across the cluster
nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Cluster Integrity
Verification Description  :  This test checks the integrity of the cluster.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  User Mask
Verification Description  :  This is a prerequisite condition to make sure the user
file creation mask (umask) is "0022".
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    0022                          0022
rac11203node1       PASSED    0022                          0022

__________________________________________________________________________________________

Verification Check        :  UDev attribtes check
Verification Description  :  This is a pre-check condition to check if the devices
entries in the Udev permissions file have been set up
correctly.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  UDev attribtes check
Verification Description  :  This is a pre-check condition to check if the devices
entries in the Udev permissions file have been set up
correctly.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Cluster Manager Integrity
Verification Description  :  This test checks the integrity of cluster manager across
the cluster nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Voting Disk
Verification Description  :  This test verifies the Oracle Clusterware voting disk
configuration which is used to determine which instances
are members of a cluster.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Time zone consistency
Verification Description  :  This task checks for the consistency of time zones across
systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Multicast check
Verification Description  :  This task checks that network interfaces in subnet are
able to communicate over multicast IP address
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________


******************************************************************************************
Clusterware recommendations
******************************************************************************************

Verification Check        :  CSS disktimeout parameter
Verification Description  :  Checks if the CSS disktimeout is set correctly on the
system
Verification Result       :  MET
Verification Summary      :  Check for CSS disktimeout parameter passed
Additional Details        :  The maximum amount of time allowed for a voting file I/O
to complete; if this time is exceeded the voting disk will
be marked as offline.  Note that this is also the amount
of time that will be required for initial cluster
formation, i.e. when no nodes have previously been up and
in a cluster.
References (URLs/Notes)   :  https://support.oracle.com/CSP/main/article?cmd=show&type=N
OT&id=294430.1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    200                           200
rac11203node1       PASSED    200                           200

__________________________________________________________________________________________

Verification Check        :  CSS reboottime parameter
Verification Description  :  Checks if the CSS reboottime is set correctly on the system
Verification Result       :  MET
Verification Summary      :  Check for CSS reboottime parameter passed
Additional Details        :  reboottime (default 3 seconds) is the amount of time
allowed for a node to complete a reboot after the CSS
daemon has been evicted.
References (URLs/Notes)   :  https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=294430.1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    3                             3
rac11203node1       PASSED    3                             3

__________________________________________________________________________________________

Verification Check        :  CSS misscount parameter
Verification Description  :  Checks if the CSS misscount is set correctly on the system
Verification Result       :  MET
Verification Summary      :  Check for CSS misscount parameter passed
Additional Details        :  The CSS misscount parameter represents the maximum time,
in seconds, that a network heartbeat can be missed before
entering into a cluster reconfiguration to evict the node
References (URLs/Notes)   :  https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=294430.1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    30                            30
rac11203node1       PASSED    30                            30

__________________________________________________________________________________________

As this is lab kit, I don’t worry about the NTP and SWAP errors-you however should!

• http://www.oracle.com/technetwork/articles/oem/raconenode-dataguard-435121.html
• http://www.oracle.com/technetwork/articles/oem/raconenode-dataguard-2-435128.html

One of my complaints with it was the limited support in OEM 11.1. At the time I was on a major consolidation project, which would have used OEM for management of the database.

OEM 11.1

Unfortunately OEM 11.1 didn’t have support for RAC One Node. Why? RON is a cluster database running on just one node. The interesting bit is that the ORACLE_SID is your normal ORACLE_SID with an underscore and a number. Under normal circumstances that number is _1, or RON_1. But as soon as you relocate the database using srvctl relocate database -d a second instance RON_2 is started until all sessions have failed over.

OEM obviously doesn’t know about RON_2: it was never discovered. Furthermore, the strict mapping of instance name to host is no longer true (the same applies for policy managed databases by the way!). A few weeks and a few switchover operations later you could be running RON_2 on racnode1.

As a consequence, the poor on-call DBA is paged about a database that has gone down, when it hasn’t-it’s up and running. As a DBA, I wouldn’t want that. After discussions with Oracle they promised to fix that problem, but it hasn’t made it into 11.1 hence this blog post about 12.

OEM 12.1 – the test

I wanted to see if that has been corrected with OEM 12.1. In order to test, I created an 11.2.0.2.0 database RON on a cluster consisting of 3 nodes, acfsprdnode1-3.

[oracle@acfsprdnode1 ~]$ srvctl config database -d RON
Database unique name: RON
Database name: RON
Oracle home: /u01/app/oracle/product/11.2.0.2
Oracle user: oracle
Spfile: +DATA/RON/spfileRON.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: RON
Database instances:
Disk Groups: DATA,FRA
Mount point paths:
Services: RON_SRV
Type: RACOneNode
Online relocation timeout: 30
Instance name prefix: RON
Candidate servers: acfsprdnode1,acfsprdnode2
Database is administrator managed

Currently the database is on node1:

[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_1 is running on node acfsprdnode1
Online relocation: INACTIVE

This is also reflected in OEM-as shown in this screenshot (click to enlarge):

Now what happens if I relocate?

[oracle@acfsprdnode1 ~]$ srvctl relocate database -d RON -n acfsprdnode2 -w 1 -v
Configuration updated to two instances
Instance RON_2 started
Services relocated
Waiting for 1 minutes for instance RON_1 to stop.....
Instance RON_1 stopped
Configuration updated to one instance
[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_2 is running on node acfsprdnode2
Online relocation: INACTIVE
[oracle@acfsprdnode1 ~]$

The relocate completes successfully, and there is only 1 instance active: RON_2 on acfsprdnode2.

What about OEM? Well to start with the database page hasn’t been updated. However, after navigating back to the home screen and to the RON database again, I have seen his:

RAC One Node after relocate

This looks good so far, need to investigate whether emails are being sent out-which is for another blog post.

But what’s really nice is that the database main page (which I set as my default), the database RON is mentioned as “UP”. Also, when I click on RON, I am directly sent to RON_2-I think that’s it!

Update

It seems I have been too happy too soon! A little more testing revealed that OEM doesn’t keep up in certain situations. After a server bounce and a little jiggery pokery I have ended up with this situation for the database:

[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_2 is running on node acfsprdnode1
Online relocation: INACTIVE

As you can see, RON_2 is now on acfsprdnode1, which hasn’t been the case before. OEM 12.1 doesn’t reflect this properly-in the below print screen shows both instances down, however the activity reports RON_2 to be up :)

Note the Summary in the upper left corner as well as the performance information in the top right. OK, all is not lost. Remember that we had a mapping of RON_2 to acfsprdnode2. So if I “assist” Clusterware in choosing the correct node, maybe I can get it back?

[oracle@acfsprdnode1 ~]$ srvctl start database -d RON -n acfsprdnode2
srvctl status database -d RON
[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_2 is running on node acfsprdnode2
Online relocation: INACTIVE
[oracle@acfsprdnode1 ~]$

By manually specifying the node to start on, it seems to work-OEM shows the instance as “up”. The ORACLE_SID seems to revert back to the one which was last used (RON_2) in this example. By specifying the node where that instance is associated with one can get the monitoring back into OEM. There is some more work to be done (or someone has to post a comment and tell me that I didn’t do it correctly!)

Summary

OEM 12.1 is a step into the right direction, and it has picked up the new instance running on the second cluster node. It seems to fail however to pick up

The operating system is Oracle Linux 5.5 running 11.2.0.2.3 and I wanted to go to 11.2.0.3.0. As you know, Grid Infrastructure upgrades are out-of-place, in other words require a separate Oracle home. This is also one of the reasons I wouldn’t want less than 20G on a non-lab like environment for the Grid Infrastructure mount points …

Now when you are upgrading from 11.2.0.x to 11.2.0.3 you need to apply a one-off patch, but the correct one! Search for patch number 12539000 (11203:ASM UPGRADE FAILED ON FIRST NODE WITH ORA-03113) and apply the one that matches your version-and pay attention to these PSUs! There is the obvious required opatch update to be performed before again as well.

So much for the prerequisites. Oracle 11.2.0.3 is available as patch 10404530, and part 3 is for Grid Infrastructure which has to be done first. This post only covers the GI upgrade, the database part is usually quite uneventful in comparison…

Upgrading Grid Infrastructure

After unzipping the third patch file you start runInstaller. But not before having carefully unset all pointers to the current 11.2.0.2 GRID_HOME (ORACLE_HOME, ORACLE_SID, LD_LIBRARY_PATH, ORA_CRS_HOME, etc)!

Clicking through OUI is mostly a matter of “next”, “next”, “next”, the action starts with the rootupgrade.sh script. Here’s the output from node1:

[root@node1 ~]# /u01/crs/11.2.0.3/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME=  /u01/crs/11.2.0.3

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/crs/11.2.0.3/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation

ASM upgrade has started on first node.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
...
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
PRCA-1056 : Unable to upgrade ACFS from version 11.2.0.2.0 to version 11.2.0.3.0
PRCT-1011 : Failed to run "advmutil". Detailed error: advmutil:
CLSU-00100: Operating System function: open64 failed with error data: 2advmutil: CLSU-00101: Operating System error message: No such file or directory|advmutil: CLSU-00103: error location: OOF_1|advmutil: CLSU-00104: additional error information: open64 (/dev/asm/orahomevol-315)|advmutil: ADVM-09006: Error opening volume /dev/asm/orahomevol-315
srvctl upgrade model -first ... failed
Failed to perform first node tasks for cluster modeling upgrade at /u01/crs/11.2.0.3/crs/install/crsconfig_lib.pm line 9088.
/u01/crs/11.2.0.3/perl/bin/perl -I/u01/crs/11.2.0.3/perl/lib -I/u01/crs/11.2.0.3/crs/install /u01/crs/11.2.0.3/crs/install/rootcrs.pl execution failed

So that was not too great indeed-my update failed halfway through. Two facts make this bearable:

1. rootupgrade.sh (and root.sh for that matter) are restartable since 11.2.0.2 at least
2. A great deal of logging is available in $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_hostname.log

Now advmutil was correct-there were no volumes in /dev/asm/*

An analysis of the rootcrs_node1.log file showed that the command that failed was this one

2012-01-06 10:09:10: Executing cmd: /u01/crs/11.2.0.3/bin/srvctl upgrade model  -s 11.2.0.2.0 -d 11.2.0.3.0 -p first
2012-01-06 10:09:12: Command output:
>  PRCA-1056 : Unable to upgrade ACFS from version 11.2.0.2.0 to version 11.2.0.3.0
>  PRCT-1011 : Failed to run "advmutil". Detailed error: advmutil: CLSU-00100: Operating System function: open64 failed with error data: 2|advmutil: CLSU-00101: Operating System error message: No such file or directory|advmutil: CLSU-00103: error location: OOF_1|advmutil: CLSU-00104: additional error information: open64 (/dev/asm/orahomevol-315)|advmutil: ADVM-09006: Error opening volume /dev/asm/orahomevol-315
>End Command output
2012-01-06 10:09:12:   "/u01/crs/11.2.0.3/bin/srvctl upgrade model  -s 11.2.0.2.0 -d 11.2.0.3.0 -p first" failed with status 1.
2012-01-06 10:09:12: srvctl upgrade model -first ... failed

Thinking Clearly

Thinking Clearly is an idea I thought I had adopted from Cary Millsap, but sadly I didn’t apply it here! Lesson learned: don’t assume, check!

I however assumed that because of the shutdown of the clusterware stack there wasn’t any Oracle software running on the node, hence there wouldn’t be an ADVM volume BY DEFINITION. Cluster down-ADVM down too.

Upon checking the log file again, I realised how wrong I was. Most of the lower stack Clusterware daemons were actually running by the time the srvctl command failed to upgrade ACFS to 11.2.0.3. So the reason for this failure had to be a different one. It quickly turned out that ALL the ACFS volumes were disabled. A quick check with asmcmd verified this:

$ asmcmd volinfo -a

Volume Name: ORAHOMEVOL
Volume Device: /dev/asm/orahomevol-315
State: DISABLED
Size (MB): 15120
Resize Unit (MB): 256
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath: /u01/app/oracle/product/11.2.0.2

OK, that explains it all-disabled volumes are obviously NOT presented in /dev/asm/. A call to “asmcmd volenable -a” sorted that problem.

Back to point 1 – rootupgrade.sh is restartable. I then switched back to the root session and started another attempt at running the script and: (drums please) it worked. Now all that was left to do was to run rootupgrade.sh on the second (and last) node. This completed successfully as well. The required patch for the ASM rolling upgrade by the way is needed there and then-the rootcrs_lastnode.log file has these lines:

2012-01-10 09:44:10: Command output:
>  Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
>  Started to upgrade the CSS.
>  Started to upgrade the CRS.
>  The CRS was successfully upgraded.
>  Oracle Clusterware operating version was successfully set to 11.2.0.3.0
>End Command output
2012-01-10 09:44:10: /u01/crs/11.2.0.3/bin/crsctl set crs activeversion ... passed
2012-01-10 09:45:10: Rolling upgrade is set to 1
2012-01-10 09:45:10: End ASM rolling upgrade
2012-01-10 09:45:10: Executing as oracle: /u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2
2012-01-10 09:45:10: Running as user oracle: /u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2
2012-01-10 09:45:10:   Invoking "/u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2" as user "oracle"
2012-01-10 09:45:10: Executing /bin/su oracle -c "/u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2"
2012-01-10 09:45:10: Executing cmd: /bin/su oracle -c "/u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2"
2012-01-10 09:45:51: Command output:
>
>  ASM upgrade has finished on last node.
>
>End Command output
2012-01-10 09:45:51: end rolling ASM upgrade in last

Note the ROLLING UPGRADE!

Summary

If your rootupgrade.sh script bails out with ADVMUTIL, check if your ACFS volumes are enabled-they most likely are not.

Well the short answer is they don’t give you an RPM, except for the XE version of the database which prompted me to do it myself. The big problem anyone faces with RPM is that the format doesn’t seem to support files larger than 2GB. Everybody knows that the Oracle database installation is > 2G which requires a little trick on our side. And the trick is not even obscure in any way as I remembered: some time ago I read an interesting article written by Frits Hoogland about cloning Oracle homes. It’s still very relevant and can be found here:

http://fritshoogland.wordpress.com/2010/07/03/cloning-your-oracle-database-software-installation/

Now that gave me the idea:

1. You install the oracle binaries on a reference host
2. Apply any patches and PSUs you need
3. Wrap the oracle home up in a tar-ball just the way Frits describes by descending into $ORACLE_HOME and creating a tar archive of all files, excluding those ending in “*.log”, network config files in $ORACLE_HOME/network/admin and anything in $ORACLE_HOME/dbs. We don’t want to proliferate our database initialisation files …
4. You make that tarball available on a central repository and export that with CIFS/NFS or whatever other mechanism you like
5. Mount this exported file system in /media, so that /media/db11.2.0.3/ has the database.tar.gz file available
6. Install the RPM

Simple! Piet de Visser would be proud.As it turned out creating (or building) the RPM is the hard part! I have spent a fair amount of time to understand it, and realised that it’s amazing! However IMO RPM is primarly aimed at packaging/building software from the source, something the oracle installer doesn’t really do in the same way.

To get you started, here’s a sample spec file I used, named Oracle11203.spec.

%define name            Oracle11203EE
# package release
%define release         1.0
# vendor release
%define version         11.2.0.3

Buildroot:              %{_topdir}/BUILDROOT/%{name}-%{release}-rootdir
Summary:                Oracle Database 11.2.0.3 EE
License:                Commercial
Name:                   %{name}
Version:                %{version}
Release:                %{release}
Group:                  Applications/Databases
Vendor:                 Martin Bach Consulting
URL:                    http://martincarstenbach.wordpress.com
Requires:               oracle-validated

%description
Oracle 11.2.0.3 Enterprise Edition installation. Requires the binaries
in tar-compressed format in /media/db11.2.0.3/database.tar.gz ready for
cloning. A good source for how to package your system up for cloning can
be found here:

http://fritshoogland.wordpress.com/2010/07/03/cloning-your-oracle-database-software-installation/

The oracle-validated is used on Oracle Linux 5 to create a sensible installation
environment. Ensure you review the environment before you build oracle!

%post
if [ -d "/u01/app/oracle/product/11.2.0.3/" ];
then echo "ORACLE_HOME directory exists-aborting";
exit 127
fi

echo "cloning the oracle home now to /u01/app/oracle/product/11.2.0.3/"
mkdir -p /u01/app/oracle/product/11.2.0.3
tar --gzip -xvf /media/db11.2.0.3/database.tar.gz -C /u01/app/oracle/product/11.2.0.3
cd /u01/app/oracle/product/11.2.0.3/clone/bin
./clone.pl ORACLE_HOME="/u01/app/oracle/product/11.2.0.3" ORACLE_BASE="/u01/app/oracle" -defaultHomeName

%files
%defattr(-,root,root,-)

How RPM normally works

When building RPMs, a certain directory structure is assumed, such as a top directory (/home/martin/rpm) and certain subdirectories:

• BUILD
• BUILDROOT
• RPMS
• SOURCES
• SPECS
• SRPMS
• tmp

The RPM build process is controlled via a “SPEC” file. Documentation of these isn’t too great, especially since the format and process has changed over time (see references). The SPEC file follows the process you would normally follow when compiling a piece of software from source. The SPEC file puts more structure around it.

Your source code goes into SOURCES (still tar-compressed). RPM can be instructed to prepare (=uncompress and patch) the tarball, which by default goes into BUILD. When running the configure command, you usually set the prefix to BUILDROOT. After the build completed (using make), you install the package in BUILDROOT. RPM then catalogs the files inside BUILDROOT and uses the defattr() clause in the SPEC file to assign permissions. This is a big relief-in earlier versions of RPM each file that went into the RPM had to be listed under the %files section.

How this process works

The above SPEC skips almost all of these steps. In our example it’s not necessary to download any sources, all the logic is contained in the SPEC file %POST section.In other words, all you need to do to build the RPM is to copy and paste the above code into the Oracle11203EE.spec file. You build the actual RPM using “rpmbuild -ba Oracle11203EE.spec”. At the end, you find a RPM file in the RPMS/x86-64 directory which can from then on be used to install Oracle on any server. And yes, the RPM is empty except for some header information and a few lines of code.

Note that the SPEC file isn’t really complete: add mount commands and error checking as you see fit. You might also want to replace literals with variables etc-the idea was to give the principle away.

Reference

• http://fedoraproject.org/wiki/How_to_create_an_RPM_package
• http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/index.html

After the engineer plugged the card in, and powered the server up I logged in as root to find out what about the pre-christmas present. I knew it was a PCI card, so surely lspci would tell me more. Here’s the output:

lspci -vvvv

41:00.0 Mass storage controller: Fusion-io ioDimm3 (v1.2) (rev 01)
Subsystem: Hewlett-Packard Company Unknown device 324d
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 10
Region 5: Memory at dfef0000 (32-bit, non-prefetchable) [size=64K]
[virtual] Expansion ROM at 98400000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
Address: 0000000000000000  Data: 0000
Capabilities: [60] Express Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 1, ExtTag+
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4

So it’s indeed a FusionIO card (an ioDrive to be precise), and it’s plugged into a x4 slot, the mimimum required. What to do next? Always good to read the docs. The fusionio.com website allows you to download documentation and drivers after a free registration. Luckily the website didn’t have kernel modules for Oracle Linux I’m using (RHEL 5 only), which gives me the opportunity to build the software from source. I don’t like surprises, therefore I created my ~/.rpmmacros with the following content:

[root@computer1 rpm]# cat ~/.rpmmacros
%_topdir /home/martin/rpm
%_tmppath /home/martin/rpm/tmp

This obviously requires the full tree underneath the topdir, namely

• BUILD
• BUILDROOT
• RPMS
• SOURCES
• SPECS
• SRPMS
• tmp

With these directories in place it’s as simple as running rpmbuild –rebuild iomemory-vsl-2.3.1.123-1.0.src.rpm as a non-root user (martin in my case) and wait for the RPMs to be created in the RPMs/x86-64 directory. Following the documentation again I installed the needed software:

[root@computer1 x86_64]# rpm -ihv iomemory-vsl-2.6.18-194.26.1.0.1.el5-2.3.1.123-1.0.x86_64.rpm
Preparing...                ########################################### [100%]
1:iomemory-vsl-2.6.18-194########################################### [100%]
[root@computer1 fusionio]# rpm -Uvh lib*.rpm
Preparing...                ########################################### [100%]
1:libfio                 ########################################### [ 50%]
2:libfusionjni           ########################################### [100%]
[root@computer1 fusionio]# rpm -Uvh fio*.rpm
Preparing...                ########################################### [100%]
1:fio-common             ########################################### [ 14%]
2:fio-util               ########################################### [ 29%]
3:fio-remote-util        ########################################### [ 43%]
4:fio-smis               ########################################### [ 57%]
5:fio-snmp-agentx        ########################################### [ 71%]
6:fio-snmp-mib           ########################################### [ 86%]
7:fio-sysvinit           ########################################### [100%]
[root@computer1 fusionio]#

Note that you don’t actually need the fio-sysvinit package if your distribution is reasonably modern. UDEV will load any drivers automatically.

With that completed as well it’s time to load the kernel module and watch Linux do the rest. The fio-status tool queries the card’s helth:

[root@computer1 x86_64]# fio-status

Found 1 ioDrive in this system
Fusion-io driver version: 2.3.1 build 123

fct0    Attached as 'fioa' (block device)
HP StorageWorks 320GB IO Accelerator, Product Number:AJ878A SN:07902
Alt PN:507152-001
PCI:41:00.0
Firmware v5.0.5, rev 43674
322.55 GBytes block device size, 396 GBytes physical device size
Sufficient power available: Unknown
Internal temperature: 40.4 degC, max 40.9 degC
Media status: Healthy; Reserves: 100.00%, warn at 10.00%

That’s it! so simple-another blog post will detail how I ran a first orion benchmark on it and show some impressive numbers.

The Oak Table Network organised “Oak Table Sunday”, a hugely successful event on Sunday afternoon. This event featured some of the brightest Oracle minds, and thanks to a very relaxed atmosphere made it all a truly exceptional experience. I have to say that the audience was quite illustrious too-I didn’t recognise Paul Vallee from Pythian with his Movember moustache at first and to my great joy I finally met Piet de Visser again. After exchanging a few words with them I ran into so many people it was just great!

Unfortunately I couldn’t make it before the HA panel session, where Alex Gorbatchev, Dave Ensor, James Morle, Greg Rahn, Dan Norris, Graham Wood, Jonathan Lewis and Mogens Norgaard and all the others I just forgot to mention answered questions from the audience.

Inevitably, the Exadata discussion started. One member of the audience put forward the proposition that there should be an Exadata simulator. While everyone in the audience agreed, feedback from the panel was different. And I tend to side with the panel on this one. Sure, Oracle is very secretive about Exadata, but I expect a business reason behind it. As everyone who implements Exadata knows, Oracle wants to use ACS to provide support. Some customers however have decided not to treat Exadata as a black box and want to understand and ultimately support it themselves. Other consultancies such as Enkitec and VX Company develop a business model to provide independent support for Exadata implementations.

However, all of the involved parties (including Oracle!) struggle to find good, experienced, reliable and dedicated staff for Exadata support. Now why could that be? First of all Exadata is a lot less accessible. Unlike other Oracle software, it is not possible to experiment with Exadata without actually getting one. And renting lab time is probably a difficult thing to do as well. As Jonathan Lewis put it, you sometimes have a base plan of what you want to measure, and quickly develop different approaches to the same problem which require setup time. If the lab time is let’s say 48 hours, then spending a couple of hours on a test case takes valuable time away. Needless to say, lab time is in high demand and it might be that you have to wait a long time to be back.

Now the counter argument is of course that you couldn’t do this with a massive p-Series server either so the problem of testing your application on the new hardware has always been the same. Except that Exadata adds a secret sauce to the processing which you cannot get elsewhere ;)

So far I fully agree. There is no way you can try Exadata features without getting one, and if the initial assessment was wrong, your benefit of migrating to Exadata might be less than what you were hoping for. I do however disagree with the idea that a simulator makes that situation any better. You won’t be able to use a VM to find out if there is a performance benefit migrating to Exadata. First of all, your laptop or whichever other hardware you are using will not have the power to simulate a cell two hex-core X5670 and 24G of RAM. Neither will you have a F20 card used as flash cache in the real cell. Also, even at the low-end of the Exadata product range the system comes with three cell servers…

Having said that I do think that it would be very nice if cell patching could be practiced outside the really precious hardware for which getting downtime is probably very difficult. Every RAC DBA has applied a bundle patch in the past, or a PSU for that matter. That’s easy, and not different on Exadata. A cell patch however is a completely different thing, especially with the added twist that a completely broken cell can have a big impact on availability and processing power. So please Oracle, do something and let us test cell patches in software!

It’s been difficult to choose the correct one for me, for now I’m working with Ubuntu 11.10. One reason is that it’s said to be user friendly, and highly customisable. It comes with all the right ingredients for running different hypervisors, including my favourite: xen.

Important update! See “Security” below.

Background on Xen

For those who don’t know the story behind xen, here’s a short summary (errors and omissions are not intentional!)

Xen allows para-virtualisation of operating systems. Unlike with hardware virtualisation you know from the likes of Virtual Box and VMware products, a para-virtualised operating system must be aware of the fact it is virtualised. This requires access to the source code, a reason why you can’t run para-virtualised Windows on Linux Xen. Microsoft has their own imitation of xen, called hyper-V but it lacks in usability and functionality.

Xen boots into a privileged hypervisor, called dom0 with exclusive access to the hardware. Guests (also known as virtual machines or domUs) interact with the hardware through the hypervisor.

After a while, xen was bought by Citrix. Xen sources initially weren’t part of the official kernel and had to be patched into it. This meant a kernel had to be compiled as well, something not everyone is comfortable doing. For one reason or antoher, Red Hat stopped doing so with Fedora 9, and Red Hat Linux 5.x is the last of their enterprise distributions to include dom0 support. Red Hat 5 is based on kernel 2.6.18.x and further back ports from upstream development. Kernel 2.6.18 is known as the “xenified kernel”, because of said patches.

At that stage it looked as if Xen was dead (at least to me). Processors were increasingly capable to perform bottlenecking operations in silicone rather than software speeding ESX up by lots. The benefits of para-virtualisation were less visible.

OpenSuSE is the only distribution carrying the xen patches forward to my knowledge; however their 11.4 distribution had lots of usability problems causing me to abandon it. OpenSuSE 12.1 seems to continue with the forward-carrying of patches.

However in years of hard work the xen patches were integrated into the mainline kernel, an interesting background story is provided by Wim Coekarts here: http://blogs.oracle.com/wim/entry/linux_mainline_contains_all_the.  The important change in kernel development was the move away from xenified kernels (which are separate from “the” kernel) to paravirtualised ops, or pvops for short. The pvops kernel contains the entire necessary infrastructure to either be a xen kernel, or “normal” desktop kernel. Fedora 16, Ubuntu and others use the PVOPS kernel, which requires xen 4.0.

With kernels 2.6.39+ for the first time it is possible to provide a dom0 and domU without the kernel patch.

My Ubuntu experience

I installed a minimal version of Ubuntu 11.10 server on my environment. To benefit from Xen virtualisation, the following packages plus their dependencies were required:

• xen-hypervisor-4.1-amd64
• xen-utils-4.1
• xenwatch
• xen-tools
• xen-utils-common
• xenstore-utils
• virtinst
• virt-viewer
• virt-manager

This creates a new menu for grub2 to boot into. The use of grub2 came as a little shock to me after I mastered lilo and “legacy” grub. To get on friendlier terms with it I installed startupmanager, and referred to help.ubuntu.com/community/Grub2, which is a great resource. A propos “install”. Being a RPM person, the use of apt-cache, dpkg and apt-get required a little getting used to. Luckily there are many debian/Ubuntu tutorials out there, including cheat sheets for these tools.

Using startupmanager I changed the default boot menu entry to my xen kernel and rebooted. Due to a bug in the ACPI implementation on the server I had to change /etc/default/grub file to acpi=ht.

Also, the /etc/xen/xend-config.sxp file needs to be amended to include (xend-unix-server yes)

Time for a reboot! After you got the console back, check that xen and virsh:

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 23149     8     r-----     16.6
# virsh version
Compiled against library: libvir 0.9.2
Using library: libvir 0.9.2
Using API: Xen 3.0.1
Running hypervisor: Xen 4.1.0

Networking

After the reboot I was surprised to see a “virbr0” interface in the output of ifconfig. I checked the /etc/network/interfaces file where Ubuntu stores its network configuration but couldn’t find any. The libvirt wiki (http://wiki.libvirt.org/page/Networking) describes the virbr0 interface as the “default” network created by libvirt at boot time, using NAT. The configuration file can be found in /usr/share/libvirt/networks/default.xml and it’s used for quick and dirty live CD or other boot media that aren’t used for production. The documentation states quite clearly that you shouldn’t manually add any interfaces to the bridge. The benefit of using the default network is the ease of use: a DHCP server distributes an IP address, and NAT works out of the box including the correct setting of firewall rules. However it’s not what I want.

What I wanted though was a setup similar to what I had before: host only networks, using br0 for the “public” network, br1 and br2 for RAC interconnect or iSCSI related storage. The tools to achieve this are in package bridge-utils. To set up a bridge, edit /etc/network/interfaces and add these directives:

auto br0
iface br0 inet static
pre-up brctl addbr br0
post-down brctl delbr br0
address 192.168.99.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

auto br1
iface br1 inet static
pre-up brctl addbr br1
post-down brctl delbr br1
address 192.168.100.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

auto br2
iface br2 inet static
pre-up brctl addbr br2
post-down brctl delbr br2
address 192.168.101.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

You could even bridge eth0 to it but that is not intended. My domUs are nicely hidden from the outside world.

More loop devices

Nearly there! I actually was in the process of adding shared storage to my server when I noticed I only had 8 loop devices available to me. Normally not a big deal – unload the “loop” module, and load it again with a different value for max_loop. However, the loop module is statically compiled into the kernel.

After a lot of fiddling and the helpful comment (see below) I managed to solve the problem by amending /etc/default/grub. It contains a default line for kernel arguments. Mine has the following content:

GRUB_CMDLINE_LINUX_DEFAULT=”max_loop=256 console=tty0 acpi=off”

I wouldn’t set acpi off if I didn’t have a problem with the server failing to reboot due to a fauly ACPI implementation. I also can’t experiment with different NUMA settings which is a shame.

Security

One thing I have forgotten to mention is the security aspect-my server didn’t have a host based firewall started right away. After trying to understand “SuSEFirewall” and IPTables I was very pleasantly surprised to learn about ufw. A few commands secure your environment, have a look at these 2 great resources for ubuntu 11.10:

• https://help.ubuntu.com/11.10/serverguide/C/firewall.html#firewall-ufw
• https://wiki.ubuntu.com/UncomplicatedFirewall

Summmary

I looks promising! Just need to work out more loop devices and I’ll be good to start installing the first operating system on the new system.

The procedure is described in the OEM 12 Cloud Control Advanced Installation and Configuration Guide, chapter 6. The process is very similar to the non-RPM based agent deployment. Let’s have a loot at it in detail.

Operations to be performed on the OMS

Log in to the OMS host and log in to OEM using emcli as shown in these exapmles (which are taken from the official Cloud Control documentation):

[oracle@oem12oms ~]$ emcli login -username=sysman
Enter password :

Login successful

[oracle@oem12oms ~]$ emcli sync
Synchronized successfully

You can of course log in with a different username, appropriate permissions assumed. The next step is to get the agent image. If you would like to enquire which agents are staged on the OMS, use the following query:

[oracle@oem12oms ~]$ emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /home/oracle/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list  ...
-----------------------------------------------
Version = 12.1.0.1.0
Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.

I installed OEM as Linux 64 bit, and it comes with the 64bit agent out of the box. You have to use self-update either in online or offline mode to add additional agents (when they become available). Now let’s get the RPM. The documentation states you have to use emcli get_agentimage_rpm. The command help is shown below:

[oracle@oem12oms ~]$ emcli help get_agentimage_rpm
emcli get_agentimage_rpm
-destination_dir="destination directory"
-platform="linux"

That looks simple enough! It nearly enough was. My first attempt is shown here.

[oracle@oem12oms tmp]$ emcli get_agentimage_rpm -destination=/tmp -platform="Linux x86-64" -version="12.1.0.1.0"
Platform:Linux x86-64
Destination:/tmp
Checking for disk space requirements...
=== Partition Detail ===
Space free : 3 GB
Space required : 1 GB
RPM creation in progress ...
Check the logs at /tmp/get_agentimage_rpm_2011-11-22_12-10-43-PM.log
Copying agent image from software library to /tmp
Setting property ORACLE_HOME to:/u01/gc12.1/oms
calling pulloneoffs with arguments:/u01/gc12.1/oms/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip12.1.0.1.0Linux x86-64/tmptrue
Agent Image copied successfully...
Creation of RPM started...
RPM creation failed ...

Oops, that didn’t work. I had a look at the log file:

[oracle@oem12oms tmp]$ cat /tmp/get_agentimage_rpm_2011-11-22_12-10-43-PM.log
Platform:Linux x86-64
Version:12.1.0.1.0
Destination:/tmp
Getting Agent Image ...
Getting Platforms list  ...
Platform Short Name:linux_x64
Platform ID:226
Path:/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip
Final zip location given by the getFullAgentImage api is:/tmp/12.1.0.1.0_PluginsOneoffs_226.zip
Image exists.
Directory /usr/lib/oracle doesnt exist. Please create the directory with write permissions and then retry the emcli command.

Well ok, a requirement for a directory “/usr/lib/oracle” on the OMS wasn’t documented but not a show stopper! After the directory has been created I tried again, and you guessed it: the RPM creation failed again. This time the log’s content read:

[oracle@oem12oms tmp]$ cat /tmp/get_agentimage_rpm_2011-11-22_12-05-15-PM.log
Platform:Linux x86-64
Version:12.1.0.1.0
Destination:/tmp
Getting Agent Image ...
Getting Platforms list  ...
Platform Short Name:linux_x64
Platform ID:226
Path:/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip
Final zip location given by the getFullAgentImage api is:/tmp/12.1.0.1.0_PluginsOneoffs_226.zip
Image exists.
Executing command:/u01/gc12.1/oms/install/rpm/rpm.sh /tmp /u01/gc12.1/oms 12.1.0.1.0_PluginsOneoffs_226.zip
/u01/gc12.1/oms/install/rpm/rpm.sh: line 53: rpmbuild: command not found
RPM creation failed...
Command : /u01/gc12.1/oms/install/rpm/rpm.sh /tmp /u01/gc12.1/oms 12.1.0.1.0_PluginsOneoffs_226.zip completed with status = 1
Deleting file:/tmp/12.1.0.1.0_PluginsOneoffs_226.zip
RPM creation failed ...

Allright then-rpmbuild wasn’t there-was it documented? Not that I recall off. Again, simple to fix. For RHEL 5 the rpmbuild script is in package “rpm-build.x86_64 : Scripts and executable programs used to build packages”. After applying this the RPM creation succeeded:

[oracle@oem12oms tmp]$ emcli get_agentimage_rpm -destination=/tmp -platform="Linux x86-64" -version="12.1.0.1.0"
Platform:Linux x86-64
Destination:/tmp
Checking for disk space requirements...
=== Partition Detail ===
Space free : 3 GB
Space required : 1 GB
RPM creation in progress ...
Check the logs at /tmp/get_agentimage_rpm_2011-11-22_12-10-43-PM.log
Copying agent image from software library to /tmp
Setting property ORACLE_HOME to:/u01/gc12.1/oms
calling pulloneoffs with arguments:/u01/gc12.1/oms/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip12.1.0.1.0Linux x86-

64/tmptrue
Agent Image copied successfully...
Creation of RPM started...
RPM creation successful.
Agent image to rpm conversion completed successfully
[oracle@oem12oms tmp]$

Great-result!

On the future managed target

I shipped the RPM to the intended memory target, oem12db.localdomin. In installed the RPM on the database host as shown here:

[oracle@oem12db tmp]$ ls -lrt
total 225316
-rw-r-----. 1 oracle oinstall 230719020 Nov 22 12:18 oracle-agt-12.1.0.1.0-1.0.x86_64.rpm
[oracle@oem12db tmp]$ su -
Password:
[root@oem12db ~]# rpm -ihv /tmp/oracle-agt-12.1.0.1.0-1.0.x86_64.rpm
Preparing...                ########################################### [100%]
Running the prereq
1:oracle-agt             ########################################### [100%]
Follow the below steps to complete the agent rpm installation:
1. Edit the properties file: /usr/lib/oracle/agent/agent.properties with the correct values
2. Execute the command /etc/init.d/config.pl
[root@oem12db ~]#

It’s a “real” RPM, albeit with a (IMO) uncommon file structure for oracle software (extracted with rpm -qil oracle-agt):

Name        : oracle-agt                   Relocations: (not relocatable)
Version     : 12.1.0.1.0                        Vendor: (none)
Release     : 1.0                           Build Date: Tue 22 Nov 2011 12:13:08 PM GMT
Install Date: Tue 22 Nov 2011 12:19:19 PM GMT      Build Host: oem12oms.localdomain
Group       : Applications/OracleAGT        Source RPM: oracle-agt-12.1.0.1.0-1.0.src.rpm
Size        : 559861217                        License: Oracle Corporation
Signature   : (none)
Packager    : Oracle <www.oracle.com>
URL         : http://www.oracle.com
Summary     : Oracle Agent 11g Enterprise Edition
Description :
Oracle Agent 11g Enterprise Edition
/etc/init.d/agentrpm
/etc/init.d/config.pl
/usr/lib/oracle/agent
...

It turned out all the files reside under /usr/lib/oracle (remember that directory I created earlier?). The RPM files won’t stay there though, /usr/lib/oracle is only a staging area as I’ll demonstrate later. The next step is to modify the agent.properties file. Update as needed, my example is shown here:

[root@oem12db ~]# cat /usr/lib/oracle/agent/agent.properties
#-------------------------------------------------------------------------------
#s_OMSHost:<String> OMS host info required to connect to OMS
#s_OMSPort:<String> OMS port info required to connect to OMS
#AGENT_REGISTRATION_PASSWORD:<String> Agent Registration Password needed to
#     establish a secure connection to the OMS.
#-------------------------------------------------------------------------------
s_OMSHost=oem12oms
s_OMSPort=4901
AGENT_REGISTRATION_PASSWORD=topsecret
#-------------------------------------------------------------------------------
#agentUserName:<String> User name with which the agent should be installed.
#agentUserGroup:<String> Group to which the agent user belogs.
#-------------------------------------------------------------------------------
agentUserName=oracle
agentUserGroup=oinstall
#-------------------------------------------------------------------------------
#OraInvLoc:<String> Absolute path of the inventory location where the agent user has write permissions.
#Example : OraInvLoc=/usr/lib/oraInventory
#-------------------------------------------------------------------------------
#OraInvLoc=#OraInvLoc#
#-------------------------------------------------------------------------------
#BASEDIR:<String> Location of the agent base directory.
#Example: BASEDIR=/tmp/agentNG
#-------------------------------------------------------------------------------
BASEDIR=/u01/app/oracle/product/agent12c
#-------------------------------------------------------------------------------
#ORACLE_HOSTNAME:<String> Virtual hostname where the agent is deployed.
#Example: ORACLE_HOSTNAME=hostname.domain
#-------------------------------------------------------------------------------
ORACLE_HOSTNAME=oms12db.localdomain

Remember that unlike the 11g agent 12c won’t create an “agent11c” (or “agent12c”) directory-hence I specified /u01/app/oracle/product/agent12c. Also, make sure you have the settings correct! I haven’t found a way to restart the agent setup, and ended removing the software via OUI (this doesn’t imply there is no way to restart the agent configuration!!!)

If you are unsure about the OMS port, grep -i oms_port in the OMS /path/to/oraInventory/logs/ directory. Mine wasn’t 4889 as it would have most likely been in 11.1, but rather 4901. Interestingly you don’t need to worry if the OMS is secured or not, the agent will discovery that.

In my experience you HAVE to make sure that your hostname is recorded in /etc/hosts, even if you are using DNS. I missed that bit in the initial attempt and ended up with an agent bound to localhost.localdomain in the emd.properties file. This can be fixed, but it’s a rather complex manual process.

Once you are happy with the settings, run /etc/init.d/config.pl. In summary, it starts a “clone” operation of the agent home to the base location you specified in agent.properties and starts the agent configuration. Log file locations for each step are indicated in the output, and in my case were in /u01/app/oracle/product/agent12c/core/12.1.0.1.0/cfgtoollogs/ and the oracle inventory/logs directory

When prompted, don’t forget to run root.sh! After the installation completed, the agent was happily communicating with my management server:

[oracle@oem12db ~]$ emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : 12.1.0.1.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.1.0
Agent Process ID  : 4844
Parent Process ID : 4788
Agent URL         : https://oem12db.localdomain:3872/emd/main/
Repository URL    : https://oem12oms.localdomain:4901/empbs/upload
Started at        : 2011-11-22 16:37:31
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2011-11-22 16:43:56
Last attempted upload                        : 2011-11-22 16:43:56
Total Megabytes of XML files uploaded so far : 0.02
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 51.31%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2011-11-22 16:43:37
Last successful heartbeat to OMS             : 2011-11-22 16:43:37

---------------------------------------------------------------
Agent is Running and Ready

You may have heard that the agent functionality is now on a “as needed” basis, i.e. the agent footprint is lower. Instead of including every possible functionality in the agent (such as in 11.1), only functionality that is really needed is added in 12c in form of plugins. This is reflected in the inventory.xml file:

...
<HOME_LIST>
<HOME NAME="agent12g0" LOC="/u01/app/oracle/product/agent12c/core/12.1.0.1.0" TYPE="O" IDX="2">
<REFHOMELIST>
<REFHOME LOC="/u01/app/oracle/product/agent12c/sbin"/>
<REFHOME LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.emas.discovery.plugin_12.1.0.1.0"/>
<REFHOME LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.xa.discovery.plugin_12.1.0.1.0"/>
<REFHOME LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.db.discovery.plugin_12.1.0.1.0"/>
<REFHOME LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.oh.discovery.plugin_12.1.0.1.0"/>
<REFHOME LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.oh.agent.plugin_12.1.0.1.0"/>
</REFHOMELIST>
</HOME>
<HOME NAME="sbin12g0" LOC="/u01/app/oracle/product/agent12c/sbin" TYPE="O" IDX="3">
<DEPHOMELIST>
<DEPHOME LOC="/u01/app/oracle/product/agent12c/core/12.1.0.1.0"/>
</DEPHOMELIST>
</HOME>
<HOME NAME="oracle_sysman_emas_12_1_0_1_0_discovery_Home0" LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.emas.discovery.plugin_12.1.0.1.0" TYPE="O" IDX="4" PLUGIN="T">
<DEPHOMELIST>
<DEPHOME LOC="/u01/app/oracle/product/agent12c/core/12.1.0.1.0"/>
</DEPHOMELIST>
</HOME>
<HOME NAME="oracle_sysman_xa_12_1_0_1_0_discovery_Home0" LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.xa.discovery.plugin_12.1.0.1.0" TYPE="O" IDX="5" PLUGIN="T">
<DEPHOMELIST>
<DEPHOME LOC="/u01/app/oracle/product/agent12c/core/12.1.0.1.0"/>
</DEPHOMELIST>
</HOME>
<HOME NAME="oracle_sysman_db_12_1_0_1_0_discovery_Home0" LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.db.discovery.plugin_12.1.0.1.0" TYPE="O" IDX="6" PLUGIN="T">
<DEPHOMELIST>
<DEPHOME LOC="/u01/app/oracle/product/agent12c/core/12.1.0.1.0"/>
</DEPHOMELIST>
</HOME>
<HOME NAME="oracle_sysman_oh_12_1_0_1_0_discovery_Home0" LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.oh.discovery.plugin_12.1.0.1.0" TYPE="O" IDX="7" PLUGIN="T">
<DEPHOMELIST>
<DEPHOME LOC="/u01/app/oracle/product/agent12c/core/12.1.0.1.0"/>
</DEPHOMELIST>
</HOME>
<HOME NAME="oracle_sysman_oh_12_1_0_1_0_agent_Home0" LOC="/u01/app/oracle/product/agent12c/plugins/oracle.sysman.oh.agent.plugin_12.1.0.1.0" TYPE="O" IDX="8" PLUGIN="T">
<DEPHOMELIST>
<DEPHOME LOC="/u01/app/oracle/product/agent12c/core/12.1.0.1.0"/>
</DEPHOMELIST>
</HOME>
</HOME_LIST>

I need to read a lot more of the documentation to find out how to add plugins to agents, and that’s for another blog post…Happy monitoring!

The downside

The use of the RPM technology here is a little peculiar-if you run “rpm -q –verify oracle-agt”, you get LOTS of missing files. It’s not really the way RPM is intended to be used, and hence I don’t see an immediate advantage of using the RPM over the agent push method.

What is the Grid Naming System?

I was initially drawn towards the GNS when it was initially released with 11.2.0.1. It is aimed at environments where the Oracle DBAs take on (yet another) piece of work, namely the DNS administration. By virtue of “subdomain delegation”, the master DNS server responsible for “example.com” hands off requests for a subdomain to this – rac.example.com – to an Oracle managed process. This was quite poorly documented initially, prompting me to figure it out myself in an earlier post: http://martincarstenbach.wordpress.com/2009/10/02/build-your-own-11-2-rac-system-part-ii-dns-dhcp-for-gns/

The problem with GNS in 11.2.0.1 was that you couldn’t really test if the DNS setup was sufficient for Oracle Installer to work, and I had a few attempts at the installation (the discussion here takes into account that I might not have been able to perform sufficient checking!)

Implementing DNS and DHCP

As a quick reminder, here is the subdomain delegation bit you add to your forward resolution zone file (with bind as the example):

$ORIGIN rac.localdomain.
@       IN      NS      gns.rac.localdomain.
gns.rac.localdomain.    IN      A       192.168.99.150

Here, I have a domain called “localdomain” for which 192.168.99.10 is the primary NS. Anything for “*.rac.localdomain” will be handed off to gns.rac.localdomain with IP address 192.168.99.150. This IP address will later be defined as the GNS VIP and as it seems, it’s the only address that must be registered in DNS. This is one of the biggest changes-with “manual” name resolution you’d add the SCAN, VIP, and public names for all cluster nodes in DNS before installation.

DHCP is very simple to set up and my example in the previously mentioned post are perfectly sufficient to get started.

Implementing on the host

The below is a sample “/etc/resolv.conf” I used – ensure that your /etc/nsswitch.conf sets “hosts” to files then DNS, which is the documented way.

options attempts: 2
options timeout: 1

search rac.localdomain localdomain
nameserver 192.168.99.150
nameserver 192.168.99.10

The “attempts” and “timeout” options were new to my builds, and the search order must include the GNS domain, before it bubbles up to the corporate domain. This doesn’t have to be a top level domain by the way, I just kept it simple.

The hostname, as defined in /etc/hosts (and NOT in DNS like I said), has to be in the DNS subdomain. I opted for rac11203gnsnode1.rac.localdomain, with IP address 192.168.99.34. Nothing else is defined in /etc/hosts, optionally you could define the private interconnect addresses in there as well.

Testing and validating the setup

The cluvfy utility has been enhanced to test GNS before you go through the installation process and finding out it all failed. Before you install, you invoke runcluvfy comp gns -precrsinst as in this example:

[oracle@rac11203gnsnode1 ~]$ runcluvfy comp gns -precrsinst -domain rac.localdomain -vip 192.168.99.150 -verbose -n rac11203gnsnode1

Verifying GNS integrity

Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "rac.localdomain" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "192.168.99.150" resolves to a valid IP address
Checking the status of GNS VIP...

GNS integrity check passed

Verification of GNS integrity was successful.

The command is fairly self explanatory: pass the subdomain you want GNS to manage and the GNS VIP, and off it goes. With my settings in the zone file (the reverse file doesn’t have any entries) it completed successfully.

The GNS integrity can also be verified post installation as in this example:

[oracle@rac11203gnsnode1 ~]$ cluvfy comp gns -postcrsinst -verbose

Verifying GNS integrity

Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "rac.localdomain" is a valid domain name
Checking if the GNS VIP belongs to same subnet as the public network...
Public network subnets "192.168.99.0" match with the GNS VIP "192.168.99.0"
Checking if the GNS VIP is a valid address...
GNS VIP "192.168.99.150" resolves to a valid IP address
Checking the status of GNS VIP...
Checking if FDQN names for domain "rac.localdomain" are reachable

GNS resolved IP addresses are reachable

GNS resolved IP addresses are reachable

GNS resolved IP addresses are reachable
Checking status of GNS resource...
Node          Running?                  Enabled?
------------  ------------------------  ------------------------
rac11203gnsnode1  yes                       yes

GNS resource configuration check passed
Checking status of GNS VIP resource...
Node          Running?                  Enabled?
------------  ------------------------  ------------------------
rac11203gnsnode1  yes                       yes

GNS VIP resource configuration check passed.

GNS integrity check passed

Verification of GNS integrity was successful.
[oracle@rac11203gnsnode1 ~]$

Post installation

The srvctl utility has a few more options as well to query the GNS state. The most comprehensive one is the “-a” flag:

[oracle@rac11203gnsnode1 ~]$ srvctl config gns -a
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5353 to connect to mDNS
GNS status: OK
Domain served by GNS: rac.localdomain
GNS version: 11.2.0.3.0
GNS VIP network: ora.net1.network

There are more granular options returning version, status etc. The most useful of these options seems to be the “-l”, listing all allocated IPs (note that 192.168.99.34 is NOT supplied by GNS, but defined in the host file). Since we only connect via VIP or SCAN this is not a problem.

For my 1 node experimental systems, these IP addresses were used:

[oracle@rac11203gnsnode1 ~]$ srvctl config gns -l
Name                 Type Value
gnsclu-scan1-vip     A    192.168.99.36
gnsclu-scan2-vip     A    192.168.99.37
gnsclu-scan3-vip     A    192.168.99.38
rac11203gnsnode1-vip A    192.168.99.35
scan                 A    192.168.99.36
scan                 A    192.168.99.37
scan                 A    192.168.99.38
[oracle@rac11203gnsnode1 ~]$

Finally {}

If found the following addition to crsctl option very useful to quickly set up a DNS test “server”-surely not something to be done in production but interesting nevertheless.

[oracle@rac11203gnsnode1 ~]$ crsctl start testdns -h

Usage:

 crsctl start testdns [-address <IP_address>] [-port <port>][-domain <GNS_domain>] [-once][-v]
 Start a test DNS listener that listens on the given address at the given port and for specified domain
Where
 IP_address IP address to be used by the listener (defaults to hostname)
 port The port on which the listener will listen. Default value is 53.
 domain The domain query for which to listen. By default, all domain queries are processed.

-once Flag indicating that DNS listener should exit after one DNS query packet is received
 -v Verbose output
[oracle@rac11203gnsnode1 ~]$

You might also consider this interesting DNS query tool:

[oracle@rac11203gnsnode1 ~]$ crsctl query dns -servers
CRS-10018: the following configuration was found on the system:
CRS-10019: There are 2 domains in search order. They are:
rac.localdomain
localdomain
CRS-10022: There are 2 name servers. They are:
192.168.99.150
192.168.99.10
CRS-10020: number of retry attempts for name lookup is: 4
CRS-10021: timeout for each name lookup is: 5
[oracle@rac11203gnsnode1 ~]$

And if you don’t like the manual page for nslookup, you could use this little command instead to interrogate your DNS setup:

[oracle@rac11203gnsnode1 ~]$ crsctl query dns -h
Usage:
 crsctl query dns -servers
 Lists the system configured DNS server, search paths, attempt and timeout values

crsctl query dns -name <name> [-dnsserver <DNS_server_address>] [-port <port>] [-attempts <attempts>] [-timeout <timeout>] [-v]
 Returns a list of addresses returned by DNS lookup of the name with the specified DNS server
Where
 name Fully qualified domain name to lookup
 DNS_server_address Address of the DNS server on which name needs to be looked up
 port Port on which DNS server is listening
 attempts Number of retry attempts
 timeout Timeout in seconds

Summary

The handling of GNS has improved greatly, and a lot more information is available about what’s happening under the covers. Unknown to many, the main Clusterware tools undergo revisions all the time, so it’s worth running crsctl commands on the various objects available with the “-h” flag to stay up to date.