Martin's Blog

RAC One Node on Oracle Enterprise Manager 12c

Martin Bach — Tue, 17 Jan 2012 15:00:49 +0000

One of the promises from Oracle for OEM 12c was improved support for Oracle RAC One Node. I have spent quite a bit of time researching RON, and wrote a little article in 2 parts about it which you can find here:

One of my complaints with it was the limited support in OEM 11.1. At the time I was on a major consolidation project, which would have used OEM for management of the database.

OEM 11.1

Unfortunately OEM 11.1 didn’t have support for RAC One Node. Why? RON is a cluster database running on just one node. The interesting bit is that the ORACLE_SID is your normal ORACLE_SID with an underscore and a number. Under normal circumstances that number is _1, or RON_1. But as soon as you relocate the database using srvctl relocate database -d a second instance RON_2 is started until all sessions have failed over.

OEM obviously doesn’t know about RON_2: it was never discovered. Furthermore, the strict mapping of instance name to host is no longer true (the same applies for policy managed databases by the way!). A few weeks and a few switchover operations later you could be running RON_2 on racnode1.

As a consequence, the poor on-call DBA is paged about a database that has gone down, when it hasn’t-it’s up and running. As a DBA, I wouldn’t want that. After discussions with Oracle they promised to fix that problem, but it hasn’t made it into 11.1 hence this blog post about 12.

OEM 12.1 – the test

I wanted to see if that has been corrected with OEM 12.1. In order to test, I created an 11.2.0.2.0 database RON on a cluster consisting of 3 nodes, acfsprdnode1-3.

[oracle@acfsprdnode1 ~]$ srvctl config database -d RON
Database unique name: RON
Database name: RON
Oracle home: /u01/app/oracle/product/11.2.0.2
Oracle user: oracle
Spfile: +DATA/RON/spfileRON.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: RON
Database instances:
Disk Groups: DATA,FRA
Mount point paths:
Services: RON_SRV
Type: RACOneNode
Online relocation timeout: 30
Instance name prefix: RON
Candidate servers: acfsprdnode1,acfsprdnode2
Database is administrator managed

Currently the database is on node1:

[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_1 is running on node acfsprdnode1
Online relocation: INACTIVE

This is also reflected in OEM-as shown in this screenshot (click to enlarge):

Now what happens if I relocate?

[oracle@acfsprdnode1 ~]$ srvctl relocate database -d RON -n acfsprdnode2 -w 1 -v
Configuration updated to two instances
Instance RON_2 started
Services relocated
Waiting for 1 minutes for instance RON_1 to stop.....
Instance RON_1 stopped
Configuration updated to one instance
[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_2 is running on node acfsprdnode2
Online relocation: INACTIVE
[oracle@acfsprdnode1 ~]$

The relocate completes successfully, and there is only 1 instance active: RON_2 on acfsprdnode2.

What about OEM? Well to start with the database page hasn’t been updated. However, after navigating back to the home screen and to the RON database again, I have seen his:

RAC One Node after relocate

This looks good so far, need to investigate whether emails are being sent out-which is for another blog post.

But what’s really nice is that the database main page (which I set as my default), the database RON is mentioned as “UP”. Also, when I click on RON, I am directly sent to RON_2-I think that’s it!

Update

It seems I have been too happy too soon! A little more testing revealed that OEM doesn’t keep up in certain situations. After a server bounce and a little jiggery pokery I have ended up with this situation for the database:

[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_2 is running on node acfsprdnode1
Online relocation: INACTIVE

As you can see, RON_2 is now on acfsprdnode1, which hasn’t been the case before. OEM 12.1 doesn’t reflect this properly-in the below print screen shows both instances down, however the activity reports RON_2 to be up :)

Note the Summary in the upper left corner as well as the performance information in the top right. OK, all is not lost. Remember that we had a mapping of RON_2 to acfsprdnode2. So if I “assist” Clusterware in choosing the correct node, maybe I can get it back?

[oracle@acfsprdnode1 ~]$ srvctl start database -d RON -n acfsprdnode2
srvctl status database -d RON
[oracle@acfsprdnode1 ~]$ srvctl status database -d RON
Instance RON_2 is running on node acfsprdnode2
Online relocation: INACTIVE
[oracle@acfsprdnode1 ~]$

By manually specifying the node to start on, it seems to work-OEM shows the instance as “up”. The ORACLE_SID seems to revert back to the one which was last used (RON_2) in this example. By specifying the node where that instance is associated with one can get the monitoring back into OEM. There is some more work to be done (or someone has to post a comment and tell me that I didn’t do it correctly!)

Summary

OEM 12.1 is a step into the right direction, and it has picked up the new instance running on the second cluster node. It seems to fail however to pick up

Beware of ACFS when upgrading to 11.2.0.3

Martin Bach — Tue, 10 Jan 2012 16:01:32 +0000

This post is about a potential pitfall when migrating from 11.2.0.x to the next point release. I stumbled over problem this one on a two node cluster.

The operating system is Oracle Linux 5.5 running 11.2.0.2.3 and I wanted to go to 11.2.0.3.0. As you know, Grid Infrastructure upgrades are out-of-place, in other words require a separate Oracle home. This is also one of the reasons I wouldn’t want less than 20G on a non-lab like environment for the Grid Infrastructure mount points …

Now when you are upgrading from 11.2.0.x to 11.2.0.3 you need to apply a one-off patch, but the correct one! Search for patch number 12539000 (11203:ASM UPGRADE FAILED ON FIRST NODE WITH ORA-03113) and apply the one that matches your version-and pay attention to these PSUs! There is the obvious required opatch update to be performed before again as well.

So much for the prerequisites. Oracle 11.2.0.3 is available as patch 10404530, and part 3 is for Grid Infrastructure which has to be done first. This post only covers the GI upgrade, the database part is usually quite uneventful in comparison…

Upgrading Grid Infrastructure

After unzipping the third patch file you start runInstaller. But not before having carefully unset all pointers to the current 11.2.0.2 GRID_HOME (ORACLE_HOME, ORACLE_SID, LD_LIBRARY_PATH, ORA_CRS_HOME, etc)!

Clicking through OUI is mostly a matter of “next”, “next”, “next”, the action starts with the rootupgrade.sh script. Here’s the output from node1:

[root@node1 ~]# /u01/crs/11.2.0.3/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/crs/11.2.0.3

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/crs/11.2.0.3/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation

ASM upgrade has started on first node.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
...
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
PRCA-1056 : Unable to upgrade ACFS from version 11.2.0.2.0 to version 11.2.0.3.0
PRCT-1011 : Failed to run "advmutil". Detailed error: advmutil:
CLSU-00100: Operating System function: open64 failed with error data: 2advmutil: CLSU-00101: Operating System error message: No such file or directory|advmutil: CLSU-00103: error location: OOF_1|advmutil: CLSU-00104: additional error information: open64 (/dev/asm/orahomevol-315)|advmutil: ADVM-09006: Error opening volume /dev/asm/orahomevol-315
srvctl upgrade model -first ... failed
Failed to perform first node tasks for cluster modeling upgrade at /u01/crs/11.2.0.3/crs/install/crsconfig_lib.pm line 9088.
/u01/crs/11.2.0.3/perl/bin/perl -I/u01/crs/11.2.0.3/perl/lib -I/u01/crs/11.2.0.3/crs/install /u01/crs/11.2.0.3/crs/install/rootcrs.pl execution failed

So that was not too great indeed-my update failed halfway through. Two facts make this bearable:

rootupgrade.sh (and root.sh for that matter) are restartable since 11.2.0.2 at least
A great deal of logging is available in $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_hostname.log

Now advmutil was correct-there were no volumes in /dev/asm/*

An analysis of the rootcrs_node1.log file showed that the command that failed was this one

2012-01-06 10:09:10: Executing cmd: /u01/crs/11.2.0.3/bin/srvctl upgrade model  -s 11.2.0.2.0 -d 11.2.0.3.0 -p first
2012-01-06 10:09:12: Command output:
>  PRCA-1056 : Unable to upgrade ACFS from version 11.2.0.2.0 to version 11.2.0.3.0
>  PRCT-1011 : Failed to run "advmutil". Detailed error: advmutil: CLSU-00100: Operating System function: open64 failed with error data: 2|advmutil: CLSU-00101: Operating System error message: No such file or directory|advmutil: CLSU-00103: error location: OOF_1|advmutil: CLSU-00104: additional error information: open64 (/dev/asm/orahomevol-315)|advmutil: ADVM-09006: Error opening volume /dev/asm/orahomevol-315
>End Command output
2012-01-06 10:09:12:   "/u01/crs/11.2.0.3/bin/srvctl upgrade model  -s 11.2.0.2.0 -d 11.2.0.3.0 -p first" failed with status 1.
2012-01-06 10:09:12: srvctl upgrade model -first ... failed

Thinking Clearly

Thinking Clearly is an idea I thought I had adopted from Cary Millsap, but sadly I didn’t apply it here! Lesson learned: don’t assume, check!

I however assumed that because of the shutdown of the clusterware stack there wasn’t any Oracle software running on the node, hence there wouldn’t be an ADVM volume BY DEFINITION. Cluster down-ADVM down too.

Upon checking the log file again, I realised how wrong I was. Most of the lower stack Clusterware daemons were actually running by the time the srvctl command failed to upgrade ACFS to 11.2.0.3. So the reason for this failure had to be a different one. It quickly turned out that ALL the ACFS volumes were disabled. A quick check with asmcmd verified this:

$ asmcmd volinfo -a

Volume Name: ORAHOMEVOL
Volume Device: /dev/asm/orahomevol-315
State: DISABLED
Size (MB): 15120
Resize Unit (MB): 256
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath: /u01/app/oracle/product/11.2.0.2

OK, that explains it all-disabled volumes are obviously NOT presented in /dev/asm/. A call to “asmcmd volenable -a” sorted that problem.

Back to point 1 – rootupgrade.sh is restartable. I then switched back to the root session and started another attempt at running the script and: (drums please) it worked. Now all that was left to do was to run rootupgrade.sh on the second (and last) node. This completed successfully as well. The required patch for the ASM rolling upgrade by the way is needed there and then-the rootcrs_lastnode.log file has these lines:

2012-01-10 09:44:10: Command output:
>  Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
>  Started to upgrade the CSS.
>  Started to upgrade the CRS.
>  The CRS was successfully upgraded.
>  Oracle Clusterware operating version was successfully set to 11.2.0.3.0
>End Command output
2012-01-10 09:44:10: /u01/crs/11.2.0.3/bin/crsctl set crs activeversion ... passed
2012-01-10 09:45:10: Rolling upgrade is set to 1
2012-01-10 09:45:10: End ASM rolling upgrade
2012-01-10 09:45:10: Executing as oracle: /u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2
2012-01-10 09:45:10: Running as user oracle: /u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2
2012-01-10 09:45:10:   Invoking "/u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2" as user "oracle"
2012-01-10 09:45:10: Executing /bin/su oracle -c "/u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2"
2012-01-10 09:45:10: Executing cmd: /bin/su oracle -c "/u01/crs/11.2.0.3/bin/asmca -silent -upgradeLocalASM -lastNode /u01/crs/11.2.0.2"
2012-01-10 09:45:51: Command output:
>
>  ASM upgrade has finished on last node.
>
>End Command output
2012-01-10 09:45:51: end rolling ASM upgrade in last

Note the ROLLING UPGRADE!

Summary

If your rootupgrade.sh script bails out with ADVMUTIL, check if your ACFS volumes are enabled-they most likely are not.

Provision Oracle RDBMS software via RPM

Martin Bach — Tue, 13 Dec 2011 16:20:38 +0000

I have always asked myself why Oracle doesn’t package their software as an RPM-surely such a large organisation has the resources to do so!

Well the short answer is they don’t give you an RPM, except for the XE version of the database which prompted me to do it myself. The big problem anyone faces with RPM is that the format doesn’t seem to support files larger than 2GB. Everybody knows that the Oracle database installation is > 2G which requires a little trick on our side. And the trick is not even obscure in any way as I remembered: some time ago I read an interesting article written by Frits Hoogland about cloning Oracle homes. It’s still very relevant and can be found here:

http://fritshoogland.wordpress.com/2010/07/03/cloning-your-oracle-database-software-installation/

Now that gave me the idea:

You install the oracle binaries on a reference host
Apply any patches and PSUs you need
Wrap the oracle home up in a tar-ball just the way Frits describes by descending into $ORACLE_HOME and creating a tar archive of all files, excluding those ending in “*.log”, network config files in $ORACLE_HOME/network/admin and anything in $ORACLE_HOME/dbs. We don’t want to proliferate our database initialisation files …
You make that tarball available on a central repository and export that with CIFS/NFS or whatever other mechanism you like
Mount this exported file system in /media, so that /media/db11.2.0.3/ has the database.tar.gz file available
Install the RPM

Simple! Piet de Visser would be proud.As it turned out creating (or building) the RPM is the hard part! I have spent a fair amount of time to understand it, and realised that it’s amazing! However IMO RPM is primarly aimed at packaging/building software from the source, something the oracle installer doesn’t really do in the same way.

To get you started, here’s a sample spec file I used, named Oracle11203.spec.

%define name            Oracle11203EE
# package release
%define release         1.0
# vendor release
%define version         11.2.0.3

Buildroot:              %{_topdir}/BUILDROOT/%{name}-%{release}-rootdir
Summary:                Oracle Database 11.2.0.3 EE
License:                Commercial
Name:                   %{name}
Version:                %{version}
Release:                %{release}
Group:                  Applications/Databases
Vendor:                 Martin Bach Consulting
URL:                    http://martincarstenbach.wordpress.com
Requires:               oracle-validated

%description
Oracle 11.2.0.3 Enterprise Edition installation. Requires the binaries
in tar-compressed format in /media/db11.2.0.3/database.tar.gz ready for
cloning. A good source for how to package your system up for cloning can
be found here:

http://fritshoogland.wordpress.com/2010/07/03/cloning-your-oracle-database-software-installation/

The oracle-validated is used on Oracle Linux 5 to create a sensible installation
environment. Ensure you review the environment before you build oracle!

%post
if [ -d "/u01/app/oracle/product/11.2.0.3/" ];
then echo "ORACLE_HOME directory exists-aborting";
exit 127
fi

echo "cloning the oracle home now to /u01/app/oracle/product/11.2.0.3/"
mkdir -p /u01/app/oracle/product/11.2.0.3
tar --gzip -xvf /media/db11.2.0.3/database.tar.gz -C /u01/app/oracle/product/11.2.0.3
cd /u01/app/oracle/product/11.2.0.3/clone/bin
./clone.pl ORACLE_HOME="/u01/app/oracle/product/11.2.0.3" ORACLE_BASE="/u01/app/oracle" -defaultHomeName

%files
%defattr(-,root,root,-)

How RPM normally works

When building RPMs, a certain directory structure is assumed, such as a top directory (/home/martin/rpm) and certain subdirectories:

BUILD
BUILDROOT
RPMS
SOURCES
SPECS
SRPMS
tmp

The RPM build process is controlled via a “SPEC” file. Documentation of these isn’t too great, especially since the format and process has changed over time (see references). The SPEC file follows the process you would normally follow when compiling a piece of software from source. The SPEC file puts more structure around it.

Your source code goes into SOURCES (still tar-compressed). RPM can be instructed to prepare (=uncompress and patch) the tarball, which by default goes into BUILD. When running the configure command, you usually set the prefix to BUILDROOT. After the build completed (using make), you install the package in BUILDROOT. RPM then catalogs the files inside BUILDROOT and uses the defattr() clause in the SPEC file to assign permissions. This is a big relief-in earlier versions of RPM each file that went into the RPM had to be listed under the %files section.

How this process works

The above SPEC skips almost all of these steps. In our example it’s not necessary to download any sources, all the logic is contained in the SPEC file %POST section.In other words, all you need to do to build the RPM is to copy and paste the above code into the Oracle11203EE.spec file. You build the actual RPM using “rpmbuild -ba Oracle11203EE.spec”. At the end, you find a RPM file in the RPMS/x86-64 directory which can from then on be used to install Oracle on any server. And yes, the RPM is empty except for some header information and a few lines of code.

Note that the SPEC file isn’t really complete: add mount commands and error checking as you see fit. You might also want to replace literals with variables etc-the idea was to give the principle away.

Reference

http://fedoraproject.org/wiki/How_to_create_an_RPM_package
http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/index.html

Getting started with FusionIO

Martin Bach — Mon, 12 Dec 2011 15:36:00 +0000

I have been lucky enough to do some work with Fusion IO cards in a blade server, soon to be followed by another set of tests on a full rack mounted server. I didn’t know exactly where model I was given, but powered my server down in eager anticipation of the events to come.

After the engineer plugged the card in, and powered the server up I logged in as root to find out what about the pre-christmas present. I knew it was a PCI card, so surely lspci would tell me more. Here’s the output:

lspci -vvvv

41:00.0 Mass storage controller: Fusion-io ioDimm3 (v1.2) (rev 01)
Subsystem: Hewlett-Packard Company Unknown device 324d
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- 
So it’s indeed a FusionIO card (an ioDrive to be precise), and it’s plugged into a x4 slot, the mimimum required. What to do next? Always good to read the docs. The fusionio.com website allows you to download documentation and drivers after a free registration. Luckily the website didn’t have kernel modules for Oracle Linux I’m using (RHEL 5 only), which gives me the opportunity to build the software from source. I don’t like surprises, therefore I created my ~/.rpmmacros with the following content:
[root@computer1 rpm]# cat ~/.rpmmacros
%_topdir /home/martin/rpm
%_tmppath /home/martin/rpm/tmp

This obviously requires the full tree underneath the topdir, namely

BUILD
BUILDROOT
RPMS
SOURCES
SPECS
SRPMS
tmp

With these directories in place it’s as simple as running rpmbuild –rebuild iomemory-vsl-2.3.1.123-1.0.src.rpm as a non-root user (martin in my case) and wait for the RPMs to be created in the RPMs/x86-64 directory. Following the documentation again I installed the needed software:
[root@computer1 x86_64]# rpm -ihv iomemory-vsl-2.6.18-194.26.1.0.1.el5-2.3.1.123-1.0.x86_64.rpm
Preparing...                ########################################### [100%]
1:iomemory-vsl-2.6.18-194########################################### [100%]
[root@computer1 fusionio]# rpm -Uvh lib*.rpm
Preparing...                ########################################### [100%]
1:libfio                 ########################################### [ 50%]
2:libfusionjni           ########################################### [100%]
[root@computer1 fusionio]# rpm -Uvh fio*.rpm
Preparing...                ########################################### [100%]
1:fio-common             ########################################### [ 14%]
2:fio-util               ########################################### [ 29%]
3:fio-remote-util        ########################################### [ 43%]
4:fio-smis               ########################################### [ 57%]
5:fio-snmp-agentx        ########################################### [ 71%]
6:fio-snmp-mib           ########################################### [ 86%]
7:fio-sysvinit           ########################################### [100%]
[root@computer1 fusionio]#

Note that you don’t actually need the fio-sysvinit package if your distribution is reasonably modern. UDEV will load any drivers automatically.
With that completed as well it’s time to load the kernel module and watch Linux do the rest. The fio-status tool queries the card’s helth:
[root@computer1 x86_64]# fio-status

Found 1 ioDrive in this system
Fusion-io driver version: 2.3.1 build 123

fct0    Attached as 'fioa' (block device)
HP StorageWorks 320GB IO Accelerator, Product Number:AJ878A SN:07902
Alt PN:507152-001
PCI:41:00.0
Firmware v5.0.5, rev 43674
322.55 GBytes block device size, 396 GBytes physical device size
Sufficient power available: Unknown
Internal temperature: 40.4 degC, max 40.9 degC
Media status: Healthy; Reserves: 100.00%, warn at 10.00%

That’s it! so simple-another blog post will detail how I ran a first orion benchmark on it and show some impressive numbers.

The cause for and against the Exadata simulator

Martin Bach — Thu, 08 Dec 2011 16:04:16 +0000

I am on my way back from the best UKOUG conference I ever attended, unfortunately a lot earlier than planned. Before I start forgetting all these great moments it is time to write them up. To make use of James Morle’s words: if you weren’t there, you lose. I couldn’t agree more!

The Oak Table Network organised “Oak Table Sunday”, a hugely successful event on Sunday afternoon. This event featured some of the brightest Oracle minds, and thanks to a very relaxed atmosphere made it all a truly exceptional experience. I have to say that the audience was quite illustrious too-I didn’t recognise Paul Vallee from Pythian with his Movember moustache at first and to my great joy I finally met Piet de Visser again. After exchanging a few words with them I ran into so many people it was just great!

Unfortunately I couldn’t make it before the HA panel session, where Alex Gorbatchev, Dave Ensor, James Morle, Greg Rahn, Dan Norris, Graham Wood, Jonathan Lewis and Mogens Norgaard and all the others I just forgot to mention answered questions from the audience.

Inevitably, the Exadata discussion started. One member of the audience put forward the proposition that there should be an Exadata simulator. While everyone in the audience agreed, feedback from the panel was different. And I tend to side with the panel on this one. Sure, Oracle is very secretive about Exadata, but I expect a business reason behind it. As everyone who implements Exadata knows, Oracle wants to use ACS to provide support. Some customers however have decided not to treat Exadata as a black box and want to understand and ultimately support it themselves. Other consultancies such as Enkitec and VX Company develop a business model to provide independent support for Exadata implementations.

However, all of the involved parties (including Oracle!) struggle to find good, experienced, reliable and dedicated staff for Exadata support. Now why could that be? First of all Exadata is a lot less accessible. Unlike other Oracle software, it is not possible to experiment with Exadata without actually getting one. And renting lab time is probably a difficult thing to do as well. As Jonathan Lewis put it, you sometimes have a base plan of what you want to measure, and quickly develop different approaches to the same problem which require setup time. If the lab time is let’s say 48 hours, then spending a couple of hours on a test case takes valuable time away. Needless to say, lab time is in high demand and it might be that you have to wait a long time to be back.

Now the counter argument is of course that you couldn’t do this with a massive p-Series server either so the problem of testing your application on the new hardware has always been the same. Except that Exadata adds a secret sauce to the processing which you cannot get elsewhere ;)

So far I fully agree. There is no way you can try Exadata features without getting one, and if the initial assessment was wrong, your benefit of migrating to Exadata might be less than what you were hoping for. I do however disagree with the idea that a simulator makes that situation any better. You won’t be able to use a VM to find out if there is a performance benefit migrating to Exadata. First of all, your laptop or whichever other hardware you are using will not have the power to simulate a cell two hex-core X5670 and 24G of RAM. Neither will you have a F20 card used as flash cache in the real cell. Also, even at the low-end of the Exadata product range the system comes with three cell servers…

Having said that I do think that it would be very nice if cell patching could be practiced outside the really precious hardware for which getting downtime is probably very difficult. Every RAC DBA has applied a bundle patch in the past, or a PSU for that matter. That’s easy, and not different on Exadata. A cell patch however is a completely different thing, especially with the added twist that a completely broken cell can have a big impact on availability and processing power. So please Oracle, do something and let us test cell patches in software!

Getting started with Xen virtualisation on Ubuntu 11.10

Martin Bach — Wed, 30 Nov 2011 17:06:17 +0000

After a long time and lots of problems I decided to abandon openSuSE 11.4 and its xen implementation in favour of the PVOPS kernel and a different distribution.

It’s been difficult to choose the correct one for me, for now I’m working with Ubuntu 11.10. One reason is that it’s said to be user friendly, and highly customisable. It comes with all the right ingredients for running different hypervisors, including my favourite: xen.

Important update! See “Security” below.

Background on Xen

For those who don’t know the story behind xen, here’s a short summary (errors and omissions are not intentional!)

Xen allows para-virtualisation of operating systems. Unlike with hardware virtualisation you know from the likes of Virtual Box and VMware products, a para-virtualised operating system must be aware of the fact it is virtualised. This requires access to the source code, a reason why you can’t run para-virtualised Windows on Linux Xen. Microsoft has their own imitation of xen, called hyper-V but it lacks in usability and functionality.

Xen boots into a privileged hypervisor, called dom0 with exclusive access to the hardware. Guests (also known as virtual machines or domUs) interact with the hardware through the hypervisor.

After a while, xen was bought by Citrix. Xen sources initially weren’t part of the official kernel and had to be patched into it. This meant a kernel had to be compiled as well, something not everyone is comfortable doing. For one reason or antoher, Red Hat stopped doing so with Fedora 9, and Red Hat Linux 5.x is the last of their enterprise distributions to include dom0 support. Red Hat 5 is based on kernel 2.6.18.x and further back ports from upstream development. Kernel 2.6.18 is known as the “xenified kernel”, because of said patches.

At that stage it looked as if Xen was dead (at least to me). Processors were increasingly capable to perform bottlenecking operations in silicone rather than software speeding ESX up by lots. The benefits of para-virtualisation were less visible.

OpenSuSE is the only distribution carrying the xen patches forward to my knowledge; however their 11.4 distribution had lots of usability problems causing me to abandon it. OpenSuSE 12.1 seems to continue with the forward-carrying of patches.

However in years of hard work the xen patches were integrated into the mainline kernel, an interesting background story is provided by Wim Coekarts here: http://blogs.oracle.com/wim/entry/linux_mainline_contains_all_the. The important change in kernel development was the move away from xenified kernels (which are separate from “the” kernel) to paravirtualised ops, or pvops for short. The pvops kernel contains the entire necessary infrastructure to either be a xen kernel, or “normal” desktop kernel. Fedora 16, Ubuntu and others use the PVOPS kernel, which requires xen 4.0.

With kernels 2.6.39+ for the first time it is possible to provide a dom0 and domU without the kernel patch.

My Ubuntu experience

I installed a minimal version of Ubuntu 11.10 server on my environment. To benefit from Xen virtualisation, the following packages plus their dependencies were required:

xen-hypervisor-4.1-amd64
xen-utils-4.1
xenwatch
xen-tools
xen-utils-common
xenstore-utils
virtinst
virt-viewer
virt-manager

This creates a new menu for grub2 to boot into. The use of grub2 came as a little shock to me after I mastered lilo and “legacy” grub. To get on friendlier terms with it I installed startupmanager, and referred to help.ubuntu.com/community/Grub2, which is a great resource. A propos “install”. Being a RPM person, the use of apt-cache, dpkg and apt-get required a little getting used to. Luckily there are many debian/Ubuntu tutorials out there, including cheat sheets for these tools.

Using startupmanager I changed the default boot menu entry to my xen kernel and rebooted. Due to a bug in the ACPI implementation on the server I had to change /etc/default/grub file to acpi=ht.

Also, the /etc/xen/xend-config.sxp file needs to be amended to include (xend-unix-server yes)

Time for a reboot! After you got the console back, check that xen and virsh:

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 23149     8     r-----     16.6
# virsh version
Compiled against library: libvir 0.9.2
Using library: libvir 0.9.2
Using API: Xen 3.0.1
Running hypervisor: Xen 4.1.0

Networking

After the reboot I was surprised to see a “virbr0” interface in the output of ifconfig. I checked the /etc/network/interfaces file where Ubuntu stores its network configuration but couldn’t find any. The libvirt wiki (http://wiki.libvirt.org/page/Networking) describes the virbr0 interface as the “default” network created by libvirt at boot time, using NAT. The configuration file can be found in /usr/share/libvirt/networks/default.xml and it’s used for quick and dirty live CD or other boot media that aren’t used for production. The documentation states quite clearly that you shouldn’t manually add any interfaces to the bridge. The benefit of using the default network is the ease of use: a DHCP server distributes an IP address, and NAT works out of the box including the correct setting of firewall rules. However it’s not what I want.

What I wanted though was a setup similar to what I had before: host only networks, using br0 for the “public” network, br1 and br2 for RAC interconnect or iSCSI related storage. The tools to achieve this are in package bridge-utils. To set up a bridge, edit /etc/network/interfaces and add these directives:

auto br0
iface br0 inet static
pre-up brctl addbr br0
post-down brctl delbr br0
address 192.168.99.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

auto br1
iface br1 inet static
pre-up brctl addbr br1
post-down brctl delbr br1
address 192.168.100.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

auto br2
iface br2 inet static
pre-up brctl addbr br2
post-down brctl delbr br2
address 192.168.101.10
netmask 255.255.255.0
bridge_stp on
bridge_maxwait 0
bridge_fd 0

You could even bridge eth0 to it but that is not intended. My domUs are nicely hidden from the outside world.

More loop devices

Nearly there! I actually was in the process of adding shared storage to my server when I noticed I only had 8 loop devices available to me. Normally not a big deal – unload the “loop” module, and load it again with a different value for max_loop. However, the loop module is statically compiled into the kernel.

After a lot of fiddling and the helpful comment (see below) I managed to solve the problem by amending /etc/default/grub. It contains a default line for kernel arguments. Mine has the following content:

GRUB_CMDLINE_LINUX_DEFAULT=”max_loop=256 console=tty0 acpi=off”

I wouldn’t set acpi off if I didn’t have a problem with the server failing to reboot due to a fauly ACPI implementation. I also can’t experiment with different NUMA settings which is a shame.

Security

One thing I have forgotten to mention is the security aspect-my server didn’t have a host based firewall started right away. After trying to understand “SuSEFirewall” and IPTables I was very pleasantly surprised to learn about ufw. A few commands secure your environment, have a look at these 2 great resources for ubuntu 11.10:

Summmary

I looks promising! Just need to work out more loop devices and I’ll be good to start installing the first operating system on the new system.

Installing OEM 12c agents in RPM format

Martin Bach — Tue, 22 Nov 2011 15:33:04 +0000

One of the questions I have always asked myself revolved around: “why doesn’t Oracle package certain software as an RPM on Linux?” Well this question has recently been answered in the form of the Oracle 12c agent. It IS possible to use an RPM based installation, although it doesn’t make 100 use of RPM. I have written this post to give you an idea what happens.

The procedure is described in the OEM 12 Cloud Control Advanced Installation and Configuration Guide, chapter 6. The process is very similar to the non-RPM based agent deployment. Let’s have a loot at it in detail.

Operations to be performed on the OMS

Log in to the OMS host and log in to OEM using emcli as shown in these exapmles (which are taken from the official Cloud Control documentation):

[oracle@oem12oms ~]$ emcli login -username=sysman
Enter password :

Login successful

[oracle@oem12oms ~]$ emcli sync
Synchronized successfully

You can of course log in with a different username, appropriate permissions assumed. The next step is to get the agent image. If you would like to enquire which agents are staged on the OMS, use the following query:

[oracle@oem12oms ~]$ emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /home/oracle/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list  ...
-----------------------------------------------
Version = 12.1.0.1.0
Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.

I installed OEM as Linux 64 bit, and it comes with the 64bit agent out of the box. You have to use self-update either in online or offline mode to add additional agents (when they become available). Now let’s get the RPM. The documentation states you have to use emcli get_agentimage_rpm. The command help is shown below:

[oracle@oem12oms ~]$ emcli help get_agentimage_rpm
emcli get_agentimage_rpm
-destination_dir="destination directory"
-platform="linux"

That looks simple enough! It nearly enough was. My first attempt is shown here.

[oracle@oem12oms tmp]$ emcli get_agentimage_rpm -destination=/tmp -platform="Linux x86-64" -version="12.1.0.1.0"
Platform:Linux x86-64
Destination:/tmp
Checking for disk space requirements...
=== Partition Detail ===
Space free : 3 GB
Space required : 1 GB
RPM creation in progress ...
Check the logs at /tmp/get_agentimage_rpm_2011-11-22_12-10-43-PM.log
Copying agent image from software library to /tmp
Setting property ORACLE_HOME to:/u01/gc12.1/oms
calling pulloneoffs with arguments:/u01/gc12.1/oms/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip12.1.0.1.0Linux x86-64/tmptrue
Agent Image copied successfully...
Creation of RPM started...
RPM creation failed ...

Oops, that didn’t work. I had a look at the log file:

[oracle@oem12oms tmp]$ cat /tmp/get_agentimage_rpm_2011-11-22_12-10-43-PM.log
Platform:Linux x86-64
Version:12.1.0.1.0
Destination:/tmp
Getting Agent Image ...
Getting Platforms list  ...
Platform Short Name:linux_x64
Platform ID:226
Path:/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip
Final zip location given by the getFullAgentImage api is:/tmp/12.1.0.1.0_PluginsOneoffs_226.zip
Image exists.
Directory /usr/lib/oracle doesnt exist. Please create the directory with write permissions and then retry the emcli command.

Well ok, a requirement for a directory “/usr/lib/oracle” on the OMS wasn’t documented but not a show stopper! After the directory has been created I tried again, and you guessed it: the RPM creation failed again. This time the log’s content read:

[oracle@oem12oms tmp]$ cat /tmp/get_agentimage_rpm_2011-11-22_12-05-15-PM.log
Platform:Linux x86-64
Version:12.1.0.1.0
Destination:/tmp
Getting Agent Image ...
Getting Platforms list  ...
Platform Short Name:linux_x64
Platform ID:226
Path:/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip
Final zip location given by the getFullAgentImage api is:/tmp/12.1.0.1.0_PluginsOneoffs_226.zip
Image exists.
Executing command:/u01/gc12.1/oms/install/rpm/rpm.sh /tmp /u01/gc12.1/oms 12.1.0.1.0_PluginsOneoffs_226.zip
/u01/gc12.1/oms/install/rpm/rpm.sh: line 53: rpmbuild: command not found
RPM creation failed...
Command : /u01/gc12.1/oms/install/rpm/rpm.sh /tmp /u01/gc12.1/oms 12.1.0.1.0_PluginsOneoffs_226.zip completed with status = 1
Deleting file:/tmp/12.1.0.1.0_PluginsOneoffs_226.zip
RPM creation failed ...

Allright then-rpmbuild wasn’t there-was it documented? Not that I recall off. Again, simple to fix. For RHEL 5 the rpmbuild script is in package “rpm-build.x86_64 : Scripts and executable programs used to build packages”. After applying this the RPM creation succeeded:

[oracle@oem12oms tmp]$ emcli get_agentimage_rpm -destination=/tmp -platform="Linux x86-64" -version="12.1.0.1.0"
Platform:Linux x86-64
Destination:/tmp
Checking for disk space requirements...
=== Partition Detail ===
Space free : 3 GB
Space required : 1 GB
RPM creation in progress ...
Check the logs at /tmp/get_agentimage_rpm_2011-11-22_12-10-43-PM.log
Copying agent image from software library to /tmp
Setting property ORACLE_HOME to:/u01/gc12.1/oms
calling pulloneoffs with arguments:/u01/gc12.1/oms/u01/gc12.1/oms/sysman/agent/12.1.0.1.0_AgentCore_226.zip12.1.0.1.0Linux x86-

64/tmptrue
Agent Image copied successfully...
Creation of RPM started...
RPM creation successful.
Agent image to rpm conversion completed successfully
[oracle@oem12oms tmp]$

Great-result!

On the future managed target

I shipped the RPM to the intended memory target, oem12db.localdomin. In installed the RPM on the database host as shown here:

[oracle@oem12db tmp]$ ls -lrt
total 225316
-rw-r-----. 1 oracle oinstall 230719020 Nov 22 12:18 oracle-agt-12.1.0.1.0-1.0.x86_64.rpm
[oracle@oem12db tmp]$ su -
Password:
[root@oem12db ~]# rpm -ihv /tmp/oracle-agt-12.1.0.1.0-1.0.x86_64.rpm
Preparing...                ########################################### [100%]
Running the prereq
1:oracle-agt             ########################################### [100%]
Follow the below steps to complete the agent rpm installation:
1. Edit the properties file: /usr/lib/oracle/agent/agent.properties with the correct values
2. Execute the command /etc/init.d/config.pl
[root@oem12db ~]#

It’s a “real” RPM, albeit with a (IMO) uncommon file structure for oracle software (extracted with rpm -qil oracle-agt):

Name        : oracle-agt                   Relocations: (not relocatable)
Version     : 12.1.0.1.0                        Vendor: (none)
Release     : 1.0                           Build Date: Tue 22 Nov 2011 12:13:08 PM GMT
Install Date: Tue 22 Nov 2011 12:19:19 PM GMT      Build Host: oem12oms.localdomain
Group       : Applications/OracleAGT        Source RPM: oracle-agt-12.1.0.1.0-1.0.src.rpm
Size        : 559861217                        License: Oracle Corporation
Signature   : (none)
Packager    : Oracle 
URL         : http://www.oracle.com
Summary     : Oracle Agent 11g Enterprise Edition
Description :
Oracle Agent 11g Enterprise Edition
/etc/init.d/agentrpm
/etc/init.d/config.pl
/usr/lib/oracle/agent
...

It turned out all the files reside under /usr/lib/oracle (remember that directory I created earlier?). The RPM files won’t stay there though, /usr/lib/oracle is only a staging area as I’ll demonstrate later. The next step is to modify the agent.properties file. Update as needed, my example is shown here:

[root@oem12db ~]# cat /usr/lib/oracle/agent/agent.properties
#-------------------------------------------------------------------------------
#s_OMSHost: OMS host info required to connect to OMS
#s_OMSPort: OMS port info required to connect to OMS
#AGENT_REGISTRATION_PASSWORD: Agent Registration Password needed to
#     establish a secure connection to the OMS.
#-------------------------------------------------------------------------------
s_OMSHost=oem12oms
s_OMSPort=4901
AGENT_REGISTRATION_PASSWORD=topsecret
#-------------------------------------------------------------------------------
#agentUserName: User name with which the agent should be installed.
#agentUserGroup: Group to which the agent user belogs.
#-------------------------------------------------------------------------------
agentUserName=oracle
agentUserGroup=oinstall
#-------------------------------------------------------------------------------
#OraInvLoc: Absolute path of the inventory location where the agent user has write permissions.
#Example : OraInvLoc=/usr/lib/oraInventory
#-------------------------------------------------------------------------------
#OraInvLoc=#OraInvLoc#
#-------------------------------------------------------------------------------
#BASEDIR: Location of the agent base directory.
#Example: BASEDIR=/tmp/agentNG
#-------------------------------------------------------------------------------
BASEDIR=/u01/app/oracle/product/agent12c
#-------------------------------------------------------------------------------
#ORACLE_HOSTNAME: Virtual hostname where the agent is deployed.
#Example: ORACLE_HOSTNAME=hostname.domain
#-------------------------------------------------------------------------------
ORACLE_HOSTNAME=oms12db.localdomain

Remember that unlike the 11g agent 12c won’t create an “agent11c” (or “agent12c”) directory-hence I specified /u01/app/oracle/product/agent12c. Also, make sure you have the settings correct! I haven’t found a way to restart the agent setup, and ended removing the software via OUI (this doesn’t imply there is no way to restart the agent configuration!!!)

If you are unsure about the OMS port, grep -i oms_port in the OMS /path/to/oraInventory/logs/ directory. Mine wasn’t 4889 as it would have most likely been in 11.1, but rather 4901. Interestingly you don’t need to worry if the OMS is secured or not, the agent will discovery that.

In my experience you HAVE to make sure that your hostname is recorded in /etc/hosts, even if you are using DNS. I missed that bit in the initial attempt and ended up with an agent bound to localhost.localdomain in the emd.properties file. This can be fixed, but it’s a rather complex manual process.

Once you are happy with the settings, run /etc/init.d/config.pl. In summary, it starts a “clone” operation of the agent home to the base location you specified in agent.properties and starts the agent configuration. Log file locations for each step are indicated in the output, and in my case were in /u01/app/oracle/product/agent12c/core/12.1.0.1.0/cfgtoollogs/ and the oracle inventory/logs directory

When prompted, don’t forget to run root.sh! After the installation completed, the agent was happily communicating with my management server:

[oracle@oem12db ~]$ emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : 12.1.0.1.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.1.0
Agent Process ID  : 4844
Parent Process ID : 4788
Agent URL         : https://oem12db.localdomain:3872/emd/main/
Repository URL    : https://oem12oms.localdomain:4901/empbs/upload
Started at        : 2011-11-22 16:37:31
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2011-11-22 16:43:56
Last attempted upload                        : 2011-11-22 16:43:56
Total Megabytes of XML files uploaded so far : 0.02
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 51.31%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2011-11-22 16:43:37
Last successful heartbeat to OMS             : 2011-11-22 16:43:37

---------------------------------------------------------------
Agent is Running and Ready

You may have heard that the agent functionality is now on a “as needed” basis, i.e. the agent footprint is lower. Instead of including every possible functionality in the agent (such as in 11.1), only functionality that is really needed is added in 12c in form of plugins. This is reflected in the inventory.xml file:

...

I need to read a lot more of the documentation to find out how to add plugins to agents, and that’s for another blog post…Happy monitoring!

The downside

The use of the RPM technology here is a little peculiar-if you run “rpm -q –verify oracle-agt”, you get LOTS of missing files. It’s not really the way RPM is intended to be used, and hence I don’t see an immediate advantage of using the RPM over the agent push method.

Simplified GNS setup for RAC 11.2.0.2 and newer

Martin Bach — Thu, 17 Nov 2011 18:08:50 +0000

One of the main problems I have seen with GNS (Grid Naming Service) installations was that you couldn’t really see if your DNS and DHCP configuration was correct until it’s been too late. This has been addressed, but it’s little known. There are a number of checks you can run before starting Oracle Universal Installer, and this post is about them.

What is the Grid Naming System?

I was initially drawn towards the GNS when it was initially released with 11.2.0.1. It is aimed at environments where the Oracle DBAs take on (yet another) piece of work, namely the DNS administration. By virtue of “subdomain delegation”, the master DNS server responsible for “example.com” hands off requests for a subdomain to this – rac.example.com – to an Oracle managed process. This was quite poorly documented initially, prompting me to figure it out myself in an earlier post: http://martincarstenbach.wordpress.com/2009/10/02/build-your-own-11-2-rac-system-part-ii-dns-dhcp-for-gns/

The problem with GNS in 11.2.0.1 was that you couldn’t really test if the DNS setup was sufficient for Oracle Installer to work, and I had a few attempts at the installation (the discussion here takes into account that I might not have been able to perform sufficient checking!)

Implementing DNS and DHCP

As a quick reminder, here is the subdomain delegation bit you add to your forward resolution zone file (with bind as the example):

$ORIGIN rac.localdomain.
@       IN      NS      gns.rac.localdomain.
gns.rac.localdomain.    IN      A       192.168.99.150

Here, I have a domain called “localdomain” for which 192.168.99.10 is the primary NS. Anything for “*.rac.localdomain” will be handed off to gns.rac.localdomain with IP address 192.168.99.150. This IP address will later be defined as the GNS VIP and as it seems, it’s the only address that must be registered in DNS. This is one of the biggest changes-with “manual” name resolution you’d add the SCAN, VIP, and public names for all cluster nodes in DNS before installation.

DHCP is very simple to set up and my example in the previously mentioned post are perfectly sufficient to get started.

Implementing on the host

The below is a sample “/etc/resolv.conf” I used – ensure that your /etc/nsswitch.conf sets “hosts” to files then DNS, which is the documented way.

options attempts: 2
options timeout: 1

search rac.localdomain localdomain
nameserver 192.168.99.150
nameserver 192.168.99.10

The “attempts” and “timeout” options were new to my builds, and the search order must include the GNS domain, before it bubbles up to the corporate domain. This doesn’t have to be a top level domain by the way, I just kept it simple.

The hostname, as defined in /etc/hosts (and NOT in DNS like I said), has to be in the DNS subdomain. I opted for rac11203gnsnode1.rac.localdomain, with IP address 192.168.99.34. Nothing else is defined in /etc/hosts, optionally you could define the private interconnect addresses in there as well.

Testing and validating the setup

The cluvfy utility has been enhanced to test GNS before you go through the installation process and finding out it all failed. Before you install, you invoke runcluvfy comp gns -precrsinst as in this example:

[oracle@rac11203gnsnode1 ~]$ runcluvfy comp gns -precrsinst -domain rac.localdomain -vip 192.168.99.150 -verbose -n rac11203gnsnode1

Verifying GNS integrity

Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "rac.localdomain" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "192.168.99.150" resolves to a valid IP address
Checking the status of GNS VIP...

GNS integrity check passed

Verification of GNS integrity was successful.

The command is fairly self explanatory: pass the subdomain you want GNS to manage and the GNS VIP, and off it goes. With my settings in the zone file (the reverse file doesn’t have any entries) it completed successfully.

The GNS integrity can also be verified post installation as in this example:

[oracle@rac11203gnsnode1 ~]$ cluvfy comp gns -postcrsinst -verbose

Verifying GNS integrity

Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "rac.localdomain" is a valid domain name
Checking if the GNS VIP belongs to same subnet as the public network...
Public network subnets "192.168.99.0" match with the GNS VIP "192.168.99.0"
Checking if the GNS VIP is a valid address...
GNS VIP "192.168.99.150" resolves to a valid IP address
Checking the status of GNS VIP...
Checking if FDQN names for domain "rac.localdomain" are reachable

GNS resolved IP addresses are reachable

GNS resolved IP addresses are reachable

GNS resolved IP addresses are reachable
Checking status of GNS resource...
Node          Running?                  Enabled?
------------  ------------------------  ------------------------
rac11203gnsnode1  yes                       yes

GNS resource configuration check passed
Checking status of GNS VIP resource...
Node          Running?                  Enabled?
------------  ------------------------  ------------------------
rac11203gnsnode1  yes                       yes

GNS VIP resource configuration check passed.

GNS integrity check passed

Verification of GNS integrity was successful.
[oracle@rac11203gnsnode1 ~]$

Post installation

The srvctl utility has a few more options as well to query the GNS state. The most comprehensive one is the “-a” flag:

[oracle@rac11203gnsnode1 ~]$ srvctl config gns -a
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5353 to connect to mDNS
GNS status: OK
Domain served by GNS: rac.localdomain
GNS version: 11.2.0.3.0
GNS VIP network: ora.net1.network

There are more granular options returning version, status etc. The most useful of these options seems to be the “-l”, listing all allocated IPs (note that 192.168.99.34 is NOT supplied by GNS, but defined in the host file). Since we only connect via VIP or SCAN this is not a problem.

For my 1 node experimental systems, these IP addresses were used:

[oracle@rac11203gnsnode1 ~]$ srvctl config gns -l
Name                 Type Value
gnsclu-scan1-vip     A    192.168.99.36
gnsclu-scan2-vip     A    192.168.99.37
gnsclu-scan3-vip     A    192.168.99.38
rac11203gnsnode1-vip A    192.168.99.35
scan                 A    192.168.99.36
scan                 A    192.168.99.37
scan                 A    192.168.99.38
[oracle@rac11203gnsnode1 ~]$

Finally {}

If found the following addition to crsctl option very useful to quickly set up a DNS test “server”-surely not something to be done in production but interesting nevertheless.

[oracle@rac11203gnsnode1 ~]$ crsctl start testdns -h

Usage:

 crsctl start testdns [-address ] [-port ][-domain ] [-once][-v]
 Start a test DNS listener that listens on the given address at the given port and for specified domain
Where
 IP_address IP address to be used by the listener (defaults to hostname)
 port The port on which the listener will listen. Default value is 53.
 domain The domain query for which to listen. By default, all domain queries are processed.

-once Flag indicating that DNS listener should exit after one DNS query packet is received
 -v Verbose output
[oracle@rac11203gnsnode1 ~]$

You might also consider this interesting DNS query tool:

[oracle@rac11203gnsnode1 ~]$ crsctl query dns -servers
CRS-10018: the following configuration was found on the system:
CRS-10019: There are 2 domains in search order. They are:
rac.localdomain
localdomain
CRS-10022: There are 2 name servers. They are:
192.168.99.150
192.168.99.10
CRS-10020: number of retry attempts for name lookup is: 4
CRS-10021: timeout for each name lookup is: 5
[oracle@rac11203gnsnode1 ~]$

And if you don’t like the manual page for nslookup, you could use this little command instead to interrogate your DNS setup:

[oracle@rac11203gnsnode1 ~]$ crsctl query dns -h
Usage:
 crsctl query dns -servers
 Lists the system configured DNS server, search paths, attempt and timeout values

crsctl query dns -name  [-dnsserver ] [-port ] [-attempts ] [-timeout ] [-v]
 Returns a list of addresses returned by DNS lookup of the name with the specified DNS server
Where
 name Fully qualified domain name to lookup
 DNS_server_address Address of the DNS server on which name needs to be looked up
 port Port on which DNS server is listening
 attempts Number of retry attempts
 timeout Timeout in seconds

Summary

The handling of GNS has improved greatly, and a lot more information is available about what’s happening under the covers. Unknown to many, the main Clusterware tools undergo revisions all the time, so it’s worth running crsctl commands on the various objects available with the “-h” flag to stay up to date.

Installing FreeNX on OpenSuSE 11.4

Martin Bach — Tue, 08 Nov 2011 15:14:21 +0000

After reading an article in one of my favourite computer magazines about FreeNX and NoMashine’s NX I was very interested to get this to work. Also, google are using NX for some developers-and if a technology is good enough for google than it can only be good enough for me as well.

Unfortunately there wasn’t an awful lot of documentation around for openSuSE 11.4 but by making best use of search engines I finally got it to work. Again, this is looking rather trivial in the post but was a lot of work finding out! Now here’s what I did.

Installing FreeNX

I found the RPMs for FreeNX in the standard SuSE repositories, but there are probably newer builds to be found here:

http://download.opensuse.org/pub/opensuse/repositories/X11:/RemoteDesktop/openSUSE_11.4/

It seems to be sufficient to install these packages on the server:

#  rpm -qa|grep -i nx
NX-3.4.0-21.1.x86_64
FreeNX-0.7.2-29.1.x86_64

Instead of a post-install action as part of the RPM installation you have to take action to configure the nx server. First, you need to create this directory tree if you haven’t got cups installed on the server (FreeNX makes use of CUPS for printer sharing).

# mkdir -p /usr/lib64/cups/backend/ipp

Now you should be able to run the configuration. One thing you will notice is a prompt to generate a SSH key. I would suggest you let it generate the key rather than using the very well-known NoMachine key. Since FreeNX relies on SSH to get access to the box it’s quite secure, theoretically. The key generated during the installation will be used to authenticate client sessions. Here is the output from the configuration session on my host with some detail removed:

nxhost:~ # nxsetup --install --clean --purge
------> It is recommended that you use the NoMachine key for
easier setup. If you answer "y", FreeNX creates a custom
KeyPair and expects you to setup your clients manually.
"N" is default and uses the NoMachine key for installation.

Do you want to use your own custom KeyPair? [y/N] y
Removing special user "nx" ...no crontab for nx
done
Removing session database ...done
Removing logfile ...done
Removing home directory of special user "nx" ...done
Removing configuration files ...done
Setting up /etc/nxserver ...done
Generating public/private dsa key pair.
Your identification has been saved in /etc/nxserver/users.id_dsa.
Your public key has been saved in /etc/nxserver/users.id_dsa.pub.
The key fingerprint is:
f0:70:91:36:ed:asdsadsadsadsad9:21:7e:d1:9a root@nxhost
The key's randomart image is:
+--[ DSA 1024]----+
|o+.. o .o |
...
| |
+-----------------+
Setting up /var/lib/nxserver/db ...done
Setting up /var/log/nxserver.log ...done
Setting up special user "nx" ...done
Adding user "nx" to group "utmp" ...done
Setting up known_hosts and authorized_keys2 ...Unique key generated; your users must install

/var/lib/nxserver/home/.ssh/client.id_dsa.key

on their computers.
done
Setting up permissions ...done
Setting up cups nxipp backend ...done

----> Testing your nxserver configuration ...
Warning: Invalid value "COMMAND_FOOMATIC=/usr/lib64/cups/driver/foomatic-ppdfile"
Users will not be able to use foomatic.
Warning: "/usr/lib64/cups/backend/smb" is not executable.
Users will not be able to enable printing.
Warning: Invalid value "CUPS_ETC=/etc/cups/"
Users will not be able to enable printing.
Warning: Invalid value "COMMAND_START_KDE=startkde"
Users will not be able to request a KDE session.
Warning: Invalid value "COMMAND_START_CDE=cdwm"
Users will not be able to request a CDE session.
Warning: Invalid value "COMMAND_SMBMOUNT=smbmount". You'll not be able to use SAMBA.
Warning: Invalid value "COMMAND_SMBUMOUNT=smbumount". You'll not be able to use SAMBA.
Warning: Invalid cupsd version of "/usr/sbin/cupsd". Need version 1.2.
Users will not be able to enable printing.

Warnings occured during config check.
To enable these features please correct the configuration file.

<---- done

----> Testing your nxserver connection ...
HELLO NXSERVER - Version 2.1.0-72 OS (GPL, using backend: 3.4.0)
<--- done

Ok, nxserver is ready.

PAM authentication enabled:
All users will be able to login with their normal passwords.

PAM authentication will be done through SSH.
Please ensure that SSHD on localhost accepts password authentication.

You can change this behaviour in the /etc/nxserver/node.conf file.

Warning: Clients will not be able to login to this server with the standard key.
Please replace /usr/NX/share/client.id_dsa.key on all clients you want
to use with /var/lib/nxserver/home/.ssh/client.id_dsa.key
and protect it accordingly.

Since 1.5.0 you need to import the correct key via the GUI.

If you really want to use the NoMachine key please remove
'/var/lib/nxserver/home/.ssh/authorized_keys2'
and then run this script with the --setup-nomachine-key parameter.
Have Fun!

Now this looked promising! If problems are reported at this stage they are usually a) related to missing CUPS and b) due to sshd not listening at 127.0.0.1.

Configuring the node

All configuration work is done in /etc/nxserver/node.conf. I had to experiment quite a while to get this right and more or less secure. All users I allow in (really myself!) have to authenticate via username and password. I haven’t moved the SSH port to <> 22 but secured SSH so that I can only get in with a key file or openVPN. That should be enough I think, and except for a number of boring VMs for Oracle RAC there isn’t anything on the box anyway. To add users to the NX database, you can use these commands:

nxserver --adduser username
nxserver --passwd username

Replace username with the real username (obviously). Oh and you need to ensure that the keys are in authorized_keys2 in sshd_config such as shown here:

# grep authorized_keys2 /etc/ssh/sshd_config
AuthorizedKeysFile     .ssh/authorized_keys2

It’s a good idea a this point to rename any $HOME/.ssh/authorized_keys file to $HOME/.ssh/authorized_keys2 and test that you can get in before continuing. Whenever a change is made to the sshd_config, you need to reload the SSH daemon to make them active.

Next, edit node.conf and set these variables (some might need commenting out)

ENABLE_PASSDB_AUTHENTICATION=”0″
ENABLE_SSH_AUTHENTICATION=”0″
ENABLE_SU_AUTHENTICATION=”1″
ENABLE_USER_DB=”1″
NX_LOG_LEVEL=5
NX_LOG_SECURE=1
NX_LOGFILE=/var/log/nxserver.log
SESSION_LOG_CLEAN=0

Refer to the comments right above the variable for their meaning. The last 4 variables only need to be set temorarily for debugging. Note that this is only an example for authentication, other uses might be more appropriate for you-this one just worked.

To allow SU authentication, the nx user must be in group “users”. The easiest way to do so is via yast, but a usermod does the same. I have run into a rather nasty problem initially related to fonts. From the client log file I got this output:

# cat
/home/martin/.nx/F-C-xxx-1001-DF87C0E8D1C786FF18EBE57A1F32AE8A/session

NXAGENT - Version 3.4.0

Copyright (C) 2001, 2010 NoMachine.
See http://www.nomachine.com/ for more information.

Info: Agent running with pid '16169'.
Session: Starting session at 'Thu Nov  3 16:17:26 2011'.
Info: Proxy running in server mode with pid '16169'.
Info: Waiting for connection from '127.0.0.1' on port '5001'.
Info: Accepted connection from '127.0.0.1'.
Warning: Connected to remote version 3.5.0 with local version 3.4.0.
Warning: Consider checking http://www.nomachine.com/ for updates.
Info: Connection with remote proxy completed.
Info: Using ADSL link parameters 512/24/1/0.
Info: Using agent parameters 5000/10/50/0/0.
Info: Using cache parameters 4/4096KB/16384KB/16384KB.
Info: Using pack method 'adaptive-7' with session 'unix-gnome'.
Info: Using ZLIB data compression 1/1/32.
Info: Using ZLIB stream compression 4/4.
Info: No suitable cache file found.
Info: Listening to X11 connections on display ':1001'.
Info: Established X client connection.
Info: Using shared memory parameters 1/1/1/4096K.
Info: Using alpha channel in render extension.
Info: Not using local device configuration changes.
Error: Aborting session with 'Could not open default font 'fixed''.
Session: Aborting session at 'Thu Nov  3 16:17:28 2011'.
Session: Session aborted at 'Thu Nov  3 16:17:28 2011'.
Warning: Signals were not blocked in process with pid '16169'.
Info: Watchdog running with pid '16186'.
Info: Waiting the watchdog process to complete.
xrdb: Connection refused
xrdb: Can't open display ':1001'

It appeared to me that the problem was related to the “fixed” font package. I could solve this only by trial and eror, until I found the terminus-font package, installed it and got running. I think I could have installed the nx client on the server as well as it seems to have gone through it’s installation directories but didn’t have to in the end. The SuSE SDB entry for FreeNX recommends the following permission change:

# chown nx /var/lib/nxserver

I can’t comment on the other items in “Setting up things” from http://en.opensuse.org/SDB:FreeNX_server#openSUSE_11.3, I haven’t set up any of these and didn’t run into problems.

Installing and configuring the client

I tried the qtnx client but it repeatedly failed on me so reverted back to the official 3.5 client from the nomachine website. This works as a treat and is available for all major platforms, including deb and rpm packages. The client is installed in /usr/NX/-and this is the reason I went for FreeNX, as it adheres more to the LSB.

One thing to remember is to get the key you generated during the nxerver setup and copy it into your client session-otherwise you will never be able to connect to your host. You need to copy everything in BEGIN … to … END (including these lines) from the key (/var/lib/nxserver/home/.ssh/client.id_dsa.key) into the key field accessible in the general settings once the wizard completed. I don’t want to reproduce the client setup here, a good introduction is found in reference [4].

Before hitting the “connect” button I suggest that you tail -f /var/log/nxserver to troubleshoot the connection. Item [3] in the reference sections answers a number of FAQs around the troubleshooting, and I also found other distribution’s documentation (like ubuntu) very useful. FreeNX really is distribution agnostic which in this case is great.

I ran into a problem where gnome wasn’t installed correctly on my NX server, a call to zypper to install the gnome pattern solved the problem. I was truly amazed with the usability of the software, it was great fun working with it.

Summary

When you have been able to create sessions without any problems, change the log level for NX server to 0 or 1set session_log_clean to 1 again to avoid your hard disk from filling up with log information you are not interested in anyway.

References

http://en.opensuse.org/SDB:FreeNX_server
http://www.susethailand.com/suseforum/index.php?action=printpage;topic=1537.0
http://openfacts2.berlios.de/wikien/index.php/BerliosProject:FreeNX_-_FAQ
http://www.ews.uiuc.edu/userguide/clrtxt/nx_setup/

An interesting problem with ext4 on Oracle Linux 5.5

Martin Bach — Fri, 04 Nov 2011 11:19:29 +0000

I have run into an interesting problem with my Red Hat 5.5 installation. Naively I assumed that ext4 has been around for a long time it would be stable. For a test I performed for a friend, I created my database files on a file system formatted with ext4 and mounted it the same way I would have mounted an ext3 file system:

$ mount | grep ext4
/dev/mapper/mpath43p1 on /u02/oradata type ext4 (rw)

Now when I tried to create a data file within a tablespace of a certain size, I got block corruption which I found very interesting. My first thought was: you must have a corruption of the file system. So I shut down all processes accessing /u02/oradata and gave the file system a thorough checking.

# umount /u02/oradata
#
# fsck.ext4 -cfv /dev/mapper/mpath43p1
e4fsck 1.41.9 (22-Aug-2009)
Checking for bad blocks (read-only test): done
/dev/mapper/mpath43p1: Updating bad block inode.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/mapper/mpath43p1: ***** FILE SYSTEM WAS MODIFIED *****

42 inodes used (0.00%)
14 non-contiguous files (33.3%)
0 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 18/14
3679655 blocks used (61.82%)
0 bad blocks
5 large files

26 regular files
7 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
33 files
#

Have a look at the command line options and make sure you understand them before letting fsck loose on your file systems! By the way if Linux tells you the file system cannot be unmounted, use “fuser -m /u02/oradata” to list all PIDs accessing the mount point.

As you can see from the output the file system was fine-which struck me as odd. I mounted it again and started my database, surely that was only a glitch.

After repeating my test, I got the same block corruption. I now had a suspicion that ext4 might be the problem. Instead of creating a tablespace with 1 8GB data file, I used 4 2 GB data files and the problem went away.

After some experimentation I found out that the magical boundary is somewhere > 3G for single data files. Here is the proof:

SQL> create tablespace WillIBeUnusable datafile '/u02/oradata/orcl/corrupt.dbf' size 3G;

Tablespace created.

SQL> !dbv file=/u02/oradata/orcl/corrupt.dbf

DBVERIFY: Release 11.2.0.2.0 - Production on Thu Nov 3 10:32:02 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/orcl/corrupt.dbf

DBVERIFY - Verification complete

Total Pages Examined         : 393216
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 127
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 393089
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3371559 (0.3371559)
$

However a 5G data file reports these:

$ dbv file=/oradata/orcl/corrupt.dbf

DBVERIFY: Release 11.2.0.2.0 - Production on Thu Nov 3 10:33:49 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/orcl/corrupt.dbf

...

Page 655357 is marked corrupt
Corrupt block relative dba: 0x0289fffd (file 10, block 655357)
Completely zero block found during dbv:

Page 655358 is marked corrupt
Corrupt block relative dba: 0x0289fffe (file 10, block 655358)
Completely zero block found during dbv:

Page 655359 is marked corrupt
Corrupt block relative dba: 0x0289ffff (file 10, block 655359)
Completely zero block found during dbv:

Page 655360 is marked corrupt
Corrupt block relative dba: 0x028a0000 (file 10, block 655360)
Completely zero block found during dbv:

DBVERIFY - Verification complete

Total Pages Examined         : 655360
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 0
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 127
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 524160
Total Pages Marked Corrupt   : 131073
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3371948 (0.3371948)
$

That’s not good. A search on supporthtml.oracle.com revealed that ext4 is supported only in 5.6 and later. At that point I stopped bothering with it since even if I worked out where the problem was, I would still be running in an unsupported configuration. The main reason I prefer not to be in “unsupported” terrain is that I don’t like Oracle support to get away by simply dismissing the service request.

The Alternative: XFS

However, as Greg Rahn, Kevin Closson and others will tell you, you shouldn’t use ext4 for databases anyway: use XFS! The latter has been substantially improved in Oracle/Red Hat Linux 6 and should be your file system of choice.

Almost forgot the hard facts:

Oracle Linux 5.5 64bit
Oracle 11.2.0.1 single instance