Martins Blog

Trying to explain complex things in simple terms

Upgrade Clusterware 10.2 to 11.2

Posted by Martin Bach on January 21, 2010

This post is about upgrading Oracle Clusterware 10.2.0.4 on Oracle Enterprise Linux 5.4 x86-64. The setup is as follows:

  • 3 node RAC system, hostnames: racupgrade1, racupgrade2, racupgrade3
  • Oracle Clusterware 10.2.0.4 with CRS bundle patch 4-(patch 8436582)
  • Oracle ASM 10.2.0.4.1, i.e. PSU 10.2.0.4.1 (patch 8576156)
  • Oracle RDBMS 10.2.0.4.1
  • Additionally I patched OPatch and applied one off patch 6079224 (“THE INSTANCE CAN NOT START DUE TO IPC SEND TIMEOUT”)

Please note that at the time of this writing CRS bundle patch 4 is superseded by Patch Set Update 2 (PSU 2) for CRS 10.2.0.4, and the latest RDBMS/ASM PSU is 10.2.0.4.3.

I use ASMLib for the ASM disks and initially bound CRS and OCR to raw devices (raw1 and raw2).

A few words about the upgrade

First of all, the CRS upgrade is a rolling upgrade regardless what you chose. Also, ASM is now running out of the Grid Infrastructure home, so any existing ASM homes will no longer be needed. Don’t worry, the OUI can upgrade ASM transparently for you. If you’re lucky and on 11g Release 1, this could actually be a rolling upgrade as well. Since I’m on 10.2, this is not an option and the whole upgrade will require downtime. I wouldn’t not allow OUI to upgrade ASM as the migration isn’t finished until ASM is 11.2 as well.

Single Client Access Name

I tried to put the use and function of the SCAN in words and failed miserably, the below is from the documentation:

“With this release, the single client access name (SCAN) is the hostname to provide for all clients connecting to the cluster. The SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS). The SCAN eliminates the need to change clients when nodes are added to or removed from the cluster. Clients using the SCAN can also access the cluster using EZCONNECT.”

There you have it. For the sake of demonstration I simply added it to /etc/hosts on all 3 hosts.

Upgrading CRS

After the download of grid infrastructure, unzip it to a convenient location and ./runInstaller as usual. Note that the 10.2.x CRS stack needs to be up and running, but I stopped ASM and RDBMS (including dbconsole). I don’t want to upgrade the database to 11.2 yet, this will be done at a later step.

welcome screen

Inevitably, there is a warning about my ASM instance but I bravely marched on:

ASM-non rolling upgrade

On the node selection screen, I selected all nodes to be upgraded. I don’t see a reason why you’d only patch a subset of cluster nodes but never mind.

Node selection

Enter the SCAN information (see above for a description)

Scan address

The prerequisites check is a lot better here-in previous versions OUI wouldn’t recognise if you changed kernel parameters. In 11.2 you not only don’t have to change the parameters yourself, but Oracle can validate a second time. So run the fixup script on the node(s) where necessary and try again. I usually ignore the warning about UnixODBC:

Prerequisites

On the summary screen click “ok” and let the installation commence. As with almost all Oracle products you’ll have to run a script as root, in this case it’s rootupgrade.sh which you need to run on all nodes.

This worked ok for me on nodes 2 and 3, but ran into a memory problem at the end on node 1 which I managed to fix. rootupgrade.sh failed in the final stage:

uccessfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful. 

racupgrade1     2010/01/20 17:37:54     /u01/crs/oracle/product/grid/cdata/racupgrade1/backup_20100120_173754.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer... 

Checking swap space: must be greater than 500 MB.   Actual 511 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
Starting Oracle Universal Installer... 

Checking swap space: 403 MB available, 500 MB required.    Failed <<<< 

Some requirement checks failed. You must fulfill these requirements before 

continuing with the installation,

Checking the rootupgrade.sh file I noticed that this error is caused within the updnodelist.pl script. All it basically does is to move the CRS=”true” flag from the 10.2 CRS home to the new 11.2 Grid Infrastructure home by calling runInstaller with the updateNodeList flag. Re-ran it and it fixed the problem as a comparison of inventory.xml on node 1 with node 2 showed.

The successful output from the final execution of rootupgrade.sh is as follows:

[root@racupgrade3 software]#  /u01/crs/oracle/product/grid/rootupgrade.sh
Running Oracle 11g root.sh script... 

The following environment variables are set as:
 ORACLE_OWNER= oracle
 ORACLE_HOME=  /u01/crs/oracle/product/grid 

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
 Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
 Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
 Copying coraenv to /usr/local/bin ... 

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-01-20 17:53:40: Parsing the host name
2010-01-20 17:53:40: Checking for super user privileges
2010-01-20 17:53:40: User has super user privileges
Using configuration parameter file: /u01/crs/oracle/product/grid/crs/install/crsconfig_params
Creating trace directory
CSS appears healthy
EVM appears healthy
CRS appears healthy
Shutting down Oracle Cluster Ready Services (CRS):
Jan 20 17:54:04.946 | INF | daemon shutting down
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.mdnsd' on 'racupgrade3'
CRS-2676: Start of 'ora.mdnsd' on 'racupgrade3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racupgrade3'
CRS-2676: Start of 'ora.gipcd' on 'racupgrade3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racupgrade3'
CRS-2676: Start of 'ora.gpnpd' on 'racupgrade3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racupgrade3'
CRS-2676: Start of 'ora.cssdmonitor' on 'racupgrade3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racupgrade3'
CRS-2672: Attempting to start 'ora.diskmon' on 'racupgrade3'
CRS-2676: Start of 'ora.diskmon' on 'racupgrade3' succeeded
CRS-2676: Start of 'ora.cssd' on 'racupgrade3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racupgrade3'
CRS-2676: Start of 'ora.ctssd' on 'racupgrade3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racupgrade3'
CRS-2676: Start of 'ora.crsd' on 'racupgrade3' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'racupgrade3'
CRS-2676: Start of 'ora.evmd' on 'racupgrade3' succeeded
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Start upgrade invoked..
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
The CSS was successfully upgraded.
Started to upgrade the OCR.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 11.2.0.1.0
racupgrade3     2010/01/20 18:00:12     /u01/crs/oracle/product/grid/cdata/racupgrade3/backup_20100120_180012.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer... 

Checking swap space: must be greater than 500 MB.   Actual 1084 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.
Starting Oracle Universal Installer... 

Checking swap space: must be greater than 500 MB.   Actual 1037 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oracle/oraInventory
'UpdateNodeList' was successful.

Success

All is well that ends well:

[oracle@racupgrade3 ~]$ /u01/crs/oracle/product/grid/bin/crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.1.0]
[oracle@racupgrade3 ~]$ /u01/crs/oracle/product/grid/bin/crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.1.0]
[oracle@racupgrade3 ~]$ /u01/crs/oracle/product/grid/bin/crsctl query crs softwareversion
Oracle Clusterware version on node [racupgrade3] is [11.2.0.1.0] 

SQL> select * from v$version; 

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

SQL> select * from v$asm_client;

GROUP_NUMBER   INSTANCE_NAME  DB_NAME   STATUS     SOFTWARE_VERSION  COMPATIBLE_VERSION
------------- --------------- --------- ---------- ----------------- ------------------
1              +ASM3          +ASM      CONNECTED  11.2.0.1.0        11.2.0.1.0
1              orcl3          orcl      CONNECTED  10.2.0.4.0        10.2.0.3.0

Even though my setup seemed to have worked, there are some notes in the documentation stating that CRS must be instructed specifically to allow running a 10g/11.1 database:

  • Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment [ID 948456.1]
  • Grid Infrastructure Installation Guide: 5.3.2 Pinning Cluster Nodes for Oracle Database Release 10.x or 11.x

Update: during the Clusterware upgrade, the pinning of the cluster nodes on which a 10.2 database runs is performed automatically. It also implies that some of the new 11.2 Clusterware features aren’t available on these nodes until all RDBMS instances are migrated to 11.2.

About these ads

8 Responses to “Upgrade Clusterware 10.2 to 11.2”

  1. Allan said

    Martin

    Out of interset did you leave your OCr on RAW or did you migrate it to be managed by ASM

  2. [...] 18-Clusterware Upgrade Story from 10.2 to 11.2 Martin Bach-Upgrade Clusterware 10.2 to 11.2 [...]

  3. Andrew said

    Hi Martin,

    Great article, just wondering for the SCAN IP address do you actually add 3 new ip beside previous VIP ?

    Thanks,

    Andrew

    • Martin said

      Indeed! Without GNS you end up with 3 SCAN IP addresses which are cluster, not node specific. Then you have the public, private and virtual IP address per host.

      With GNS enabled you get an additional “virtual” IP that the GNS daemon uses. This one is assigned during the Grid Infrastructure installation.

  4. drew said

    Hi Martin,

    Thanks. I end up adding the SCAN IP in each node. In production we still considering if going to utilize this since adding DNS or GNS required more changes during upgrade 10gr2 to 11gr2 and unfortunately our downtime only 3 hours for 2 Node with 6 instances :(

    • Martin said

      Hi there,

      I don’t recommend putting the scan ip into the hosts file. The change to DNS can be done way ahead of time and doesn’t affect downtime at all.

      If you are in the UK, you might want to come to the next RAC and HA SIG where I am presenting about the upgrade to 11.2. This is going to be one of the items on my agenda.

  5. vemula said

    Hi martin,
    I am upgarding from 11.1.0.7 to 11.2.0.1 on RAC database . we are getting error
    1. We got error while upgrading Grid infrastructure 11.2.0.1,the error message is
    INS-40718: Single Client Access Name (SCAN) name: string could not be resolved.

    2.We tried to add SCAN Name in /etc/hosts, but got the error
    INS-40719: IP address configured for Single Client Access Name (SCAN): string is already assigned to another system.

    3.The installer could not find the IP addresses you selected as SCAN addresses. This may be because they are assigned to another system, or the IP addresses are not listed in the DNS or hosts files as assigned to this domain name.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 2,399 other followers

%d bloggers like this: