This post is about upgrading Oracle Clusterware 10.2.0.4 on Oracle Enterprise Linux 5.4 x86-64. The setup is as follows:
- 3 node RAC system, hostnames: racupgrade1, racupgrade2, racupgrade3
- Oracle Clusterware 10.2.0.4 with CRS bundle patch 4-(patch 8436582)
- Oracle ASM 10.2.0.4.1, i.e. PSU 10.2.0.4.1 (patch 8576156)
- Oracle RDBMS 10.2.0.4.1
- Additionally I patched OPatch and applied one off patch 6079224 (“THE INSTANCE CAN NOT START DUE TO IPC SEND TIMEOUT”)
Please note that at the time of this writing CRS bundle patch 4 is superseded by Patch Set Update 2 (PSU 2) for CRS 10.2.0.4, and the latest RDBMS/ASM PSU is 10.2.0.4.3.
I use ASMLib for the ASM disks and initially bound CRS and OCR to raw devices (raw1 and raw2).
A few words about the upgrade
First of all, the CRS upgrade is a rolling upgrade regardless what you chose. Also, ASM is now running out of the Grid Infrastructure home, so any existing ASM homes will no longer be needed. Don’t worry, the OUI can upgrade ASM transparently for you. If you’re lucky and on 11g Release 1, this could actually be a rolling upgrade as well. Since I’m on 10.2, this is not an option and the whole upgrade will require downtime. I wouldn’t not allow OUI to upgrade ASM as the migration isn’t finished until ASM is 11.2 as well.
Single Client Access Name
I tried to put the use and function of the SCAN in words and failed miserably, the below is from the documentation:
“With this release, the single client access name (SCAN) is the hostname to provide for all clients connecting to the cluster. The SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS). The SCAN eliminates the need to change clients when nodes are added to or removed from the cluster. Clients using the SCAN can also access the cluster using EZCONNECT.”
There you have it. For the sake of demonstration I simply added it to /etc/hosts on all 3 hosts.
Upgrading CRS
After the download of grid infrastructure, unzip it to a convenient location and ./runInstaller as usual. Note that the 10.2.x CRS stack needs to be up and running, but I stopped ASM and RDBMS (including dbconsole). I don’t want to upgrade the database to 11.2 yet, this will be done at a later step.
Inevitably, there is a warning about my ASM instance but I bravely marched on:
On the node selection screen, I selected all nodes to be upgraded. I don’t see a reason why you’d only patch a subset of cluster nodes but never mind.
Enter the SCAN information (see above for a description)
The prerequisites check is a lot better here-in previous versions OUI wouldn’t recognise if you changed kernel parameters. In 11.2 you not only don’t have to change the parameters yourself, but Oracle can validate a second time. So run the fixup script on the node(s) where necessary and try again. I usually ignore the warning about UnixODBC:
On the summary screen click “ok” and let the installation commence. As with almost all Oracle products you’ll have to run a script as root, in this case it’s rootupgrade.sh which you need to run on all nodes.
This worked ok for me on nodes 2 and 3, but ran into a memory problem at the end on node 1 which I managed to fix. rootupgrade.sh failed in the final stage:
uccessfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. racupgrade1 2010/01/20 17:37:54 /u01/crs/oracle/product/grid/cdata/racupgrade1/backup_20100120_173754.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 511 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful. Starting Oracle Universal Installer... Checking swap space: 403 MB available, 500 MB required. Failed <<<< Some requirement checks failed. You must fulfill these requirements before continuing with the installation,
Checking the rootupgrade.sh file I noticed that this error is caused within the updnodelist.pl script. All it basically does is to move the CRS=”true” flag from the 10.2 CRS home to the new 11.2 Grid Infrastructure home by calling runInstaller with the updateNodeList flag. Re-ran it and it fixed the problem as a comparison of inventory.xml on node 1 with node 2 showed.
The successful output from the final execution of rootupgrade.sh is as follows:
[root@racupgrade3 software]# /u01/crs/oracle/product/grid/rootupgrade.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/crs/oracle/product/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying dbhome to /usr/local/bin ... The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying oraenv to /usr/local/bin ... The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying coraenv to /usr/local/bin ... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2010-01-20 17:53:40: Parsing the host name 2010-01-20 17:53:40: Checking for super user privileges 2010-01-20 17:53:40: User has super user privileges Using configuration parameter file: /u01/crs/oracle/product/grid/crs/install/crsconfig_params Creating trace directory CSS appears healthy EVM appears healthy CRS appears healthy Shutting down Oracle Cluster Ready Services (CRS): Jan 20 17:54:04.946 | INF | daemon shutting down Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. Shutdown has begun. The daemons should exit soon. LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting CRS-2672: Attempting to start 'ora.mdnsd' on 'racupgrade3' CRS-2676: Start of 'ora.mdnsd' on 'racupgrade3' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'racupgrade3' CRS-2676: Start of 'ora.gipcd' on 'racupgrade3' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'racupgrade3' CRS-2676: Start of 'ora.gpnpd' on 'racupgrade3' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racupgrade3' CRS-2676: Start of 'ora.cssdmonitor' on 'racupgrade3' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'racupgrade3' CRS-2672: Attempting to start 'ora.diskmon' on 'racupgrade3' CRS-2676: Start of 'ora.diskmon' on 'racupgrade3' succeeded CRS-2676: Start of 'ora.cssd' on 'racupgrade3' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'racupgrade3' CRS-2676: Start of 'ora.ctssd' on 'racupgrade3' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'racupgrade3' CRS-2676: Start of 'ora.crsd' on 'racupgrade3' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'racupgrade3' CRS-2676: Start of 'ora.evmd' on 'racupgrade3' succeeded clscfg: EXISTING configuration version 5 detected. clscfg: version 5 is 11g Release 2. Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Start upgrade invoked.. Started to upgrade the Oracle Clusterware. This operation may take a few minutes. Started to upgrade the CSS. The CSS was successfully upgraded. Started to upgrade the OCR. Started to upgrade the CRS. The CRS was successfully upgraded. Successfully upgraded the Oracle Clusterware. Oracle Clusterware operating version was successfully set to 11.2.0.1.0 racupgrade3 2010/01/20 18:00:12 /u01/crs/oracle/product/grid/cdata/racupgrade3/backup_20100120_180012.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 1084 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful. Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 1037 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.
Success
All is well that ends well:
[oracle@racupgrade3 ~]$ /u01/crs/oracle/product/grid/bin/crsctl query crs releaseversion Oracle High Availability Services release version on the local node is [11.2.0.1.0] [oracle@racupgrade3 ~]$ /u01/crs/oracle/product/grid/bin/crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.1.0] [oracle@racupgrade3 ~]$ /u01/crs/oracle/product/grid/bin/crsctl query crs softwareversion Oracle Clusterware version on node [racupgrade3] is [11.2.0.1.0] SQL> select * from v$version; BANNER -------------------------------------------------------------------------------- Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production PL/SQL Release 11.2.0.1.0 - Production CORE 11.2.0.1.0 Production TNS for Linux: Version 11.2.0.1.0 - Production NLSRTL Version 11.2.0.1.0 - Production SQL> select * from v$asm_client; GROUP_NUMBER INSTANCE_NAME DB_NAME STATUS SOFTWARE_VERSION COMPATIBLE_VERSION ------------- --------------- --------- ---------- ----------------- ------------------ 1 +ASM3 +ASM CONNECTED 11.2.0.1.0 11.2.0.1.0 1 orcl3 orcl CONNECTED 10.2.0.4.0 10.2.0.3.0
Even though my setup seemed to have worked, there are some notes in the documentation stating that CRS must be instructed specifically to allow running a 10g/11.1 database:
- Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment [ID 948456.1]
- Grid Infrastructure Installation Guide: 5.3.2 Pinning Cluster Nodes for Oracle Database Release 10.x or 11.x
Update: during the Clusterware upgrade, the pinning of the cluster nodes on which a 10.2 database runs is performed automatically. It also implies that some of the new 11.2 Clusterware features aren’t available on these nodes until all RDBMS instances are migrated to 11.2.
Martins Blog 
Responses
Martin
Out of interset did you leave your OCr on RAW or did you migrate it to be managed by ASM
For this test I left them on raw, but have the strong intend to follow http://download.oracle.com/docs/cd/E11882_01/rac.112/e10717/votocr.htm#CHDJDFIC and move them to ASM. I need to do some more testing since our disk groups are created with external redundancy (as probably the majority of them are), which limits the number of copies we can have.
Also, very importantly, you need an ASM diskgroup with compatible.asm set to 11.2 for this to work!
[…] 18-Clusterware Upgrade Story from 10.2 to 11.2 Martin Bach-Upgrade Clusterware 10.2 to 11.2 […]
Hi Martin,
Great article, just wondering for the SCAN IP address do you actually add 3 new ip beside previous VIP ?
Thanks,
Andrew
Indeed! Without GNS you end up with 3 SCAN IP addresses which are cluster, not node specific. Then you have the public, private and virtual IP address per host.
With GNS enabled you get an additional “virtual” IP that the GNS daemon uses. This one is assigned during the Grid Infrastructure installation.
Hi Martin,
Thanks. I end up adding the SCAN IP in each node. In production we still considering if going to utilize this since adding DNS or GNS required more changes during upgrade 10gr2 to 11gr2 and unfortunately our downtime only 3 hours for 2 Node with 6 instances :(
Hi there,
I don’t recommend putting the scan ip into the hosts file. The change to DNS can be done way ahead of time and doesn’t affect downtime at all.
If you are in the UK, you might want to come to the next RAC and HA SIG where I am presenting about the upgrade to 11.2. This is going to be one of the items on my agenda.
Hi martin,
I am upgarding from 11.1.0.7 to 11.2.0.1 on RAC database . we are getting error
1. We got error while upgrading Grid infrastructure 11.2.0.1，the error message is
INS-40718: Single Client Access Name (SCAN) name: string could not be resolved.
2.We tried to add SCAN Name in /etc/hosts, but got the error
INS-40719: IP address configured for Single Client Access Name (SCAN): string is already assigned to another system.
3.The installer could not find the IP addresses you selected as SCAN addresses. This may be because they are assigned to another system, or the IP addresses are not listed in the DNS or hosts files as assigned to this domain name.