Category Archives: RAC Book

Pro Oracle Database 11g RAC on Linux is available for Kindle

Addmittedly I haven’t checked for a little while, but an email by my co-author Steve Show prompted me to go to the Amazon website and look it up.

And yes, it’s reality! Our book is now finally available as a kindle version, how great is that?!?

There isn’t really a lot more to say about this subject. I’ll wonder how many techies are intersted in the kindle version after the PDF has been out for quite a while. If you read this and decide to get the kindle version, could you please let me know how you liked it?  Personally I think the book is well suited for the Amazon reader as it’s mostly text which suits the device well.

Happy reading!


Summer Seminars

I am doing a couple of one day seminars with Oracle University, currently planned for Austria and Switzerland. They go by the title “Grid Intrastructure and Database High Availability Deep Dive”, and can be accessed via these links.

To save you from having to get the abstract, I copied it from the Oracle University website:

Providing a highly available database architecture fit for today’s fast changing requirements can be a complex task. Many technologies are available to provide resilience, each with its own advantages and possible disadvantages. This seminar begins with an overview of available HA technologies (hard and soft partitioning of servers, cold failover clusters, RAC and RAC One Node) and complementary tools and techniques to provide recovery from site failure (Data Guard or storage replication).

In the second part of the seminar, we look at Grid Infrastructure in great detail. Oracle Grid Infrastructure is the latest incarnation of the Clusterware HA framework which successfully powers every single 10g and 11g RAC installation. Despite its widespread implementation, many of its features are still not well understood by its users. We focus on Grid Infrastructure, what it is, what it does and how it can be put to best use, including the creation of an active/passive cold failover cluster for web and database resources. Special focus will be placed on the various storage options (Cluster File System, ASM, etc), the cluster interconnect and other implementation choices and on troubleshooting Grid Infrastructure. In the final part of the seminar, we explore Real Application Clusters and its various uses, from HA to scalability to consolidation. We discuss patching and workload management, coding for RAC and other techniques that will allow users to maximise the full potential of the package.

See you there if you are interested!

RAC One Node and Database Protection

An email from fellow Oak Table Member James Morle about RAC One Node and failover got me thinking about the capabilities of the product. I have written about RAC One Node (RON) in earlier posts, but haven’t really explored what happens with session failover during a database relocation.

Note this post is now showing it’s age, I initially published it when RAC One Node was a new product introduced with 11g Release2 . If you hit this article via a search engine you should keep in mind that RAC One Node (and everything else) has been greatly enhanced and this article might not apply anymore.


So to clarify what happens during failover situations I have developed a simple scenario. Taking a RON database and a service, I modified both to suit my test needs. While I was connected to the service I performed a database relocation to see what happens. Next I killed the instance (I wasn’t able to reboot the node) to simulate what happens when the node crashes.

Continue reading

Troubleshooting on an 8 node cluster

It seems I am doing a lot of fixing broken stuff recently. So this time I have been asked to repair a broken 8 node RAC cluster on OEL 5.5 with Oracle RAC The system has been moved into a different, more secure network, and its firewalls prevented all access to the machines except for ILO. Another way of “security through obscurity”. The new network didn’t allow any clients to connect to any of the 8 node RAC which means that it is actually quite expensive kit to sit idle. The cluster is not in production, it’s still being build to specification but this accessibility problem has been a holdup to the project for a little while now. Yesterday has been a breakthrough-the netops team found an error to their configuration and for the first time the hosts could be accessed via ssh. Unfortunately for me that access is possible via audited gateways using PowerBroker to which I don’t have access. Continue reading

Getting up and running with Universal Connection Pool


Martin Nash (@mpnsh) kindly pointed out a mistake on the ONS configuration in the resource element below. Thanks for that!


This post was written for 11g Release 2 and Oracle Linux 5. Where applicable I have added information on how to do this with 12c Release 1 and Oracle Linux 7.2.


Oracle’s next generation connection pooling solution, Universal Connection Pool, can be a bit tricky to set up. This is especially true when a JNDI data source is to be used-most example don’t assume such a scenario. A lot of information is out there on the net, but no one seems to have given the full picture. During the research for chapter 11 of “Pro Oracle Database 11g RAC on Linux” I learned this the hard way. Since the book has been published, a few minor changes changes have been made to the software I used at the time, and those merit an update.

Please note that this article’s emphasis is to get  this example running-it is by no means meant to be secure enough for a production release! You need to harden  the setup considerably for production, but it serves well for demonstration purposes (only).


I have used a four node RAC system as the source for my data. A 2 node cluster database with service “TAFTEST” runs on nodes 1 and 2. It’s administrator-managed and the service has both nodes set aside as “preferred” nodes. The database nodes run Oracle Enterprise Linux 5.564 bit with RAC For the sake of simplicity, I used my Windows laptop to host the Tomcat instance, which is now updated to version 6.0.30. I am using apache Ant to build the application. The current stable ant build is 1.8.2. My JDK is also upgraded to the latest and greatest, version 1.6.0_23. I am using the 32bit client package to supply me with ons.jar, ojdbc6.jar and ucp.jar.

The Oracle Linux 7.2 system uses the following components:

  • Oracle Linux 7.2 64bit
  • ant-1.9.2-9.el7.noarch
  • tomcat-7.0.54-8.el7_2.noarch including tomcat-admin-webapps-7.0.54-8.el7_2.noarch for the web interface
  • Oracle client located in /u01/app/oracle/product/
  • Owned by “oracle” for convenience

Continue reading

5 out of 5 for Pro Oracle Database 11g RAC on Linux!

For a long time I didn’t dare look at Amazon reviews for “Pro Oracle Database 11g RAC on Linux”, the book Steve Shaw and I worked on for the greatest part of the last 18 months. Well I actually joined him in March this year to be fair.

So now Steve sent me an email to notify me of some great reviews on the Amazon USA website. How cool was this? Have a look at the screenshot I took:

I would have never thought of such great reviews.

  • one of the “must have” books on Oracle RAC 11g, November 13, 2010
  • Excellent Resource for RAC Admins, November 18, 2010
  • Just when you start to think that no one book can “have it all”, December 4, 2010
  • Another RAC Gem by Steve Shaw and Martin Bach., December 14, 2010

Thank you all so much for this and purchasing the book. The link to the book and the reviews (very thorough and worth reading!) is here:

Oracle RAC One Node revisited –

Since we published the RAC book, Oracle has released patchset Amongst other things, this improved the RAC One Node option, exactly the way we expected.

How it was – the base release

A quick recap on the product as it was in RAC One Node is part of Oracle Enterprise Edition, any other software editions are explicitly not allowed. Another restriction exists for 3rd party Clusterware: it’s not allowed to use one. RAC One Node is a hybrid between full blown RAC and the active/passive cluster. The option uses Grid Infrastructure for cluster management and storage provisioning via ASM. The RAC One instance starts its life as a RAC database, limited to only one cluster node. It only ever runs on one node, but that node can change. It is strongly recommended to create a service for that RAC database. Utilities such as raconeinit provide a text based command line interface to transform that database to a “RAC One Node”-instance. In the process, the administrator can elect which nodes should be allowed to run the instance. The “omotion” utilities allowed the DBA to move the RAC One Node instance from the current node to another one. Optionally a time threshold could be set after which all ongoing transactions were to move to the new node. This feature required TAF or FAN to be set up correctly. The raconestatus utility allowed you to view the status of your RAC One Node instances. Conversion to full RAC was made possible by the racone2rac utility.

If you were after a Data Guard setup you’d be disappointed: that wasn’t supported. (The good news is: from onwards, Data Guard can be used)

So all in all, that seemed a little premature. A patch to be downloaded and applied, no Data Guard and a new set of utilities are not really user friendly. Plus, initially this patch was available for Linux only. But at least a MOS note (which I didn’t find until after having finished writing this!) exists, RAC One — Changes in [ID 1232802.1]


Instead of having to apply patch 9004119 to your environment, RAC One Node is available “out of the box” with Sadly, the Oracle RAC One Node manual has not been updated, and searches on Metalink reveal no new information. One interesting piece of information: the patch for RAC One Node is listed as “undocumented Oracle Server” section.

The creation of a RAC One Node instance has been greatly simplified-dbca has added support for it, both from the command line for silent installations as well as the interactive GUI. Consider these options for dbca.

$ dbca -help
dbca  [-silent | -progressOnly | -customCreate] {<command> <options> }  |
 { [<command> [options] ] -responseFile  <response file > }
 [-continueOnNonFatalErrors <true | false>]
Please refer to the manual for details.
You can enter one of the following command:

Create a database by specifying the following parameters:
 -templateName <name of an existing  template>
 -gdbName <global database name>
 -RACOneNodeServiceName  <Service name for the service to be
 created for RAC One Node database.>]
 [-policyManaged | -adminManaged <Policy managed or Admin managed
 Database, default is Admin managed
 [-createServerPool <To create ServerPool which will be used by the
 database to be created>]
 [-force <To create serverpool by force when adequate free servers
 are not available. This may affect already running database>]
 -serverPoolName <One serverPool Name in case of create server pool and
 comma separated list of serverPool name in case of
 use serverpool>
 -[cardinality <Specify cardinality for new serverPool to be created,
 default is the number of qualified nodes>]
 [-sid <database system identifier prefix>]

With RAC One Node you will most likely end up with a policy managed database in the end, I can’t see how an admin managed database made sense.

The srvctl command line tool has been improved to deal with the RAC One node. The most important operations are to add, remove, config and status. The nice thing about dbca is that it actually registers the database in the OCR. Immediately after the installation, you see this status information:

$ srvctl status database -d rontest
Instance rontest_1 is running on node node2
Online relocation: INACTIVE

$ srvctl config database -d rontest
Database unique name: rontest
Database name:
Oracle home: /data/oracle/product/
Oracle user: oracle
Spfile: + DATA/rontest/spfilerontest.ora
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: rontest
Database instances:
Disk Groups: DATA
Mount point paths:
Services: rontestsrv
Type: RACOneNode
Online relocation timeout: 30
Instance name prefix: rontest
Candidate servers: node2,node3
Database is administrator managed

Note that the instance_name, although the instance is administrator managed, changed to $ORACLE_SID_1. Relocating works now with the srvctl relocate database command as in this example:

$ srvctl relocate database -d rontest -n node2

You’ll get feedback about this in the output of the “status” command:

$ srvctl status database -d rontest
Instance rontest_1 is running on node node2
Online relocation: ACTIVE
Source instance: rontest_1 on node2
Destination instance: rontest_2 on node3

After the command completed, check the status again:

srvctl status database -d rontest
Instance rontest_2 is running on node node2
Online relocation: INACTIVE

The important difference between an admin managed database and a policy managed database is that you are responsible for undo tablespaces. If you don’t create and configure undo tablespaces, the relocate command will fail:

$ srvctl relocate database –d rontest -n node3                       <
PRCD-1222 : Online relocation of database rontest failed but database was restored to its original state
PRCD-1129 : Failed to start instance rontest_2 for database rontest
PRCR-1064 : Failed to start resource ora.rontest.db on node node3
CRS-5017: The resource action "ora.rontest.db start" encountered the following error:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-30013: undo tablespace 'UNDOTBS1' is currently in use
Process ID: 1587
Session ID: 35 Serial number: 1

CRS-2674: Start of 'ora.rontest.db' on 'node3' failed

In this case, the database runs on the same node. Check the ORACLE_SID (rontest_2 in my case) and modify the initialisation parameter.

SQL> select tablespace_name from dba_data_files where tablespace_name like ‘%UNDO%’;


So the tablespace was there, but the initialisation parameter was wrong! Let’s correct this:

SQL> alter system set undo_tablespace='UNDOTBS1' sid='rontest_1';

System altered.

SQL> alter system set undo_tablespace='UNDOTBS2' sid='rontest_2';

System altered.

Now the relocate will succeed.

To wrap this article up, the srvctl convert database command will convert between single instance, RAC One Node and RAC databases.

Kindle version of Pro Oracle Database 11g RAC on Linux

I had a few questions from readers whether or not there was going to be a kindle version of Pro Oracle Database 11g RAC on Linux.

The good news for those waiting is: yes! But it might take a couple of weeks for it to be released.

I checked with Jonathan Gennick who expertly oversaw the whole project and he confirmed that Amazon have been contacted to provide a kindle version.

As soon as I hear more, I’ll post it here.

Pro Oracle Database 11g RAC on Linux is out

So it has finally happened, the day many of you who patiently waited for. Amazon and other publishers are fulfilling pre-orders! The book I helped publish, Pro Oracle Database 11g RAC on Linux is out in print.

I received my author copies last week, and was very happy to see that after the book was out in the USA first, the “pre-order now” recommendation has been removed from Amazon’s UK and German websites when I last checked.

I have to give great credit to all of those who helped me make this possible with extra input and pointers to related information. On the Linux part Steve has done an incredible job compiling all this useful information you don’t find in this form anywhere else.

From now on, feedback and new research with specific reference to the book will appear on my wordpress blog (the one you are reading now), in a separate category-“RAC Book”.

So thanks again for all your patience while we were working very hard to finish the book.