Martins Blog

Trying to explain complex things in simple terms

Applying PSU 11.2.0.1.1 to Grid Infrastructure & RDBMS

Posted by Martin Bach on April 30, 2010

WARNING: The patch number for the PSU has changed, but I didn’t have time to update the blog entry yet! Please ensure you read MOS note 1082394.1 now, and disregard the instructions below. The new PSU for Grid Infrastructure is patch 9778840!

DO NOT FOLLOW THESE INSRTUCTIONS! Update to follow

So here I am again, PSU 11.2.0.1.1 for Grid Infrastructure is out, so let’s get it applied. It’s known as patch 9343627 internally. Quite a chunky download at around 340 MB!

Going through the readme we find the ususal suspects, amongst them the new opatch deployment and the locking/unlocking of the Grid Infrastructure home.

OPatch 11.2

We need opatch 11.2 for this patch-it’s patch 6880880 on metalink. Make sure to download the 11.2 version, some 26MB in size. Deploy as follows to the RDBMS home:

[oracle@rac11gr2node1 ~]$ . oraenv
ORACLE_SID = [oracle] ? admindb
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is /u01/app/oracle
[oracle@rac11gr2node1 ~]$ cd $ORACLE_HOME
[oracle@rac11gr2node1 dbhome_1]$ mv OPatch OPatch.11.1
[oracle@rac11gr2node1 dbhome_1]$ cp /mnt/db11.2/p6880880_112000_Linux-x86-64.zip .
[oracle@rac11gr2node1 dbhome_1]$ unzip -q p6880880_112000_Linux-x86-64.zip
rm [oracle@rac11gr2node1 dbhome_1]$ rm p6880880_112000_Linux-x86-64.zip

Deployment to the Grid Infrastructure home has to wait until the home is unlocked, see below. Don’t forget to deploy OPatch 11.2 though! The interesting bit about this patch is that there are parts for the RDBMS as well. And as it turned out, the zipfile for patch 9343627 contains the RDBMS PSU 1 as well. How odd-haven’t seen this before but nevermind.

Both PSUs can be applied in a rolling fashion-just ensure you stop and unlock the ORACLE_HOMEs on one node at a time. My Grid Infrastructure installation is owned by user “grid”, the RDBMS by user “oracle”. I am not using shared Oracle homes. The OS is RHEL 5.5 64bit.

Preparing the ORACLE Homes

For the RDBMS part of the patch, we need to execute a few steps as the RDBMS software owner-oracle in my case. Do this on one node at a time for a rolling patch, i.e. proceed with all the steps here and repeat on the other nodes. The big exception is if your RDBMS home is on a shared file system.

[oracle@rac11gr2node1 dbhome_1]$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_1
[oracle@rac11gr2node1 dbhome_1]$ $ORACLE_HOME/bin/srvctl stop home -o $ORACLE_HOME -s /tmp/stat_rdbms -n rac11gr2node1
[oracle@rac11gr2node1 dbhome_1]$

The stat_rdbms file lists all the resources that were up at the time of the stop command and can start them when we are done. Unzip patch 9343627 to a temporary location NOT CONTAINING the unzipped RDBMS PSU (patch 9352237) if you have it stored centrally. Like I said, PSU 1 for GI includes the RDBMS portion as well and you might risk conflicts when unzipping.

As the RDBMS software owner run the prepatch.sh script, in my case that’s in ./custom/server/9343627/custom/scripts/prepatch.sh.

[oracle@rac11gr2node1 scripts]$ ./prepatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_1

Completed fairly quickly… nothing special to report. Now as the owner of the Grid Infrastructure installation, prepare the local home for patching.

[oracle@rac11gr2node1 scripts]$ su - grid
Password:
[grid@rac11gr2node1 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/grid/product/11.2.0/crs is /u01/app/oracle
[grid@rac11gr2node1 ~]$ $ORACLE_HOME/bin/srvctl stop home -o $ORACLE_HOME -s /tmp/stat_grid -n rac11gr2node1
[grid@rac11gr2node1 ~]$

Careful-don’t overwrite the RDBMS home stat file :)

Grid Infrastructure homes are locked down and need to be unlocked before patching. Become root and unlock the home, which will shut down Clusterware daemons and bring down the node from a user’s point of view:

[grid@rac11gr2node1 ~]$ su -
Password:
[root@rac11gr2node1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/grid/product/11.2.0/crs is /u01/app/oracle
[root@rac11gr2node1 ~]# $ORACLE_HOME/crs/install/rootcrs.pl -unlock
2010-04-30 12:14:13: Parsing the host name
2010-04-30 12:14:13: Checking for super user privileges
2010-04-30 12:14:13: User has super user privileges
Using configuration parameter file: /u01/app/grid/product/11.2.0/crs/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac11gr2node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac11gr2node1'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac11gr2node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac11gr2node1'
CRS-2677: Stop of 'ora.asm' on 'rac11gr2node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac11gr2node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac11gr2node1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac11gr2node1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac11gr2node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac11gr2node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac11gr2node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac11gr2node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac11gr2node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac11gr2node1'
CRS-2677: Stop of 'ora.cssd' on 'rac11gr2node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac11gr2node1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'rac11gr2node1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac11gr2node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac11gr2node1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac11gr2node1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac11gr2node1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'rac11gr2node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac11gr2node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /u01/app/grid/product/11.2.0/crs

I had all the Grid Infrastructure services still running on nodes 2-4 which was good-so this is a “rolling patch”.

File System

Now to confuse people some more we need to switch back to the Grid Infrastructure owner and deploy OPatch 11.2:

[root@rac11gr2node1 ~]# su - grid
[grid@rac11gr2node1 ~]$ cd /u01/app/grid/product/11.2.0/crs
[grid@rac11gr2node1 crs]$ mv OPatch/ OPatch.11.1
[grid@rac11gr2node1 crs]$ cp /mnt/db11.2/p6880880_112000_Linux-x86-64.zip .
[grid@rac11gr2node1 crs]$ unzip p6880880_112000_Linux-x86-64.zip && rm p6880880_112000_Linux-x86-64.zip

I checked some more only to see that PSU 1 for GI requires 4.4 GB free storage. Bugger – resized my /u01 to accomodate:

[root@rac11gr2node2 product]# df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/root_vg-root_lv
 2.8G  2.4G  298M  89% /
 /dev/xvda1             99M   14M   81M  15% /boot
 tmpfs                 2.0G  168M  1.9G   9% /dev/shm
 /dev/mapper/oracle_vg-orabin_lv
 15G   11G  4.0G  73% /u01
 192.168.99.10:/m/download
 1.4T  125G  1.2T  10% /mnt
 [root@rac11gr2node2 product]# lvresize --size +2G /dev/mapper/oracle_vg-orabin_lv
 Extending logical volume orabin_lv to 17.00 GB
 Logical volume orabin_lv successfully resized
 [root@rac11gr2node2 product]# resize2fs /dev/mapper/oracle_vg-orabin_lv
 resize2fs 1.39 (29-May-2006)
 Filesystem at /dev/mapper/oracle_vg-orabin_lv is mounted on /u01; on-line resizing required
 Performing an on-line resize of /dev/mapper/oracle_vg-orabin_lv to 4455424 (4k) blocks.
 The filesystem on /dev/mapper/oracle_vg-orabin_lv is now 4455424 blocks long.

 [root@rac11gr2node2 product]# df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/root_vg-root_lv
 2.8G  2.4G  298M  90% /
 /dev/xvda1             99M   14M   81M  15% /boot
 tmpfs                 2.0G  168M  1.9G   9% /dev/shm
 /dev/mapper/oracle_vg-orabin_lv
 17G   11G  5.8G  64% /u01
 192.168.99.10:/m/download
 1.4T  125G  1.2T  10% /mnt
 [root@rac11gr2node2 product]#

… and that of course on all nodes…

I am now ready to apply both PSUs-recording as we are going along, let’s brave the storm!

[grid@rac11gr2node1 ~]$ echo $ORACLE_HOME $ORACLE_SID
/u01/app/grid/product/11.2.0/crs +ASM1
[grid@rac11gr2node1 ~]$ cd /mnt/db11.2/
[grid@rac11gr2node1 db11.2]$ pwd
/mnt/db11.2
[grid@rac11gr2node1 db11.2]$ ls -l
total 3717704
drwxr-xr-x 5 1000 users       4096 Apr 15 19:28 9343627
drwxrwxr-x 5 1000 users       4096 Mar 25 07:05 9352237
drwxr-xr-x 8 1000 users       4096 Aug 20  2009 database
drwxr-xr-x 8 1000 users       4096 Aug 20  2009 grid
-rw-r--r-- 1 1000 users 1239269270 Apr 26 05:30 linux.x64_11gR2_database_1of2.zip
-rw-r--r-- 1 1000 users 1111416131 Apr 26 05:30 linux.x64_11gR2_database_2of2.zip
-rw-r--r-- 1 1000 users 1052897657 Apr 26 05:29 linux.x64_11gR2_grid.zip
-rw-r--r-- 1 1000 users   27511537 Apr 30 11:47 p6880880_112000_Linux-x86-64.zip
-rw-r--r-- 1 1000 users  359993395 Apr 23 21:39 p9343627_112010_Linux-x86-64.zip
-rw-r--r-- 1 1000 users   12064953 Apr 13 08:11 p9352237_112010_Linux-x86-64.zip

Check for inventory corruption-always a good idea to do before patching:

[grid@rac11gr2node1 db11.2]$ /u01/app/grid/product/11.2.0/crs/OPatch/opatch lsinv
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/grid/product/11.2.0/crs
Central Inventory : /u01/app/oraInventory
 from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/grid/product/11.2.0/crs/oui
Log file location : /u01/app/grid/product/11.2.0/crs/cfgtoollogs/opatch/opatch2010-04-30_12-27-18PM.log

Patch history file: /u01/app/grid/product/11.2.0/crs/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/grid/product/11.2.0/crs/cfgtoollogs/opatch/lsinv/lsinventory2010-04-30_12-27-18PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes
 Local node = rac11gr2node1
 Remote node = rac11gr2node2
 Remote node = rac11gr2node3
 Remote node = rac11gr2node4

--------------------------------------------------------------------------------

OPatch succeeded.

Now GO!

/u01/app/grid/product/11.2.0/crs/OPatch/opatch napply -local -oh /u01/app/grid/product/11.2.0/crs -id 9343627,9352237
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/grid/product/11.2.0/crs
Central Inventory : /u01/app/oraInventory
 from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/grid/product/11.2.0/crs/oui
Log file location : /u01/app/grid/product/11.2.0/crs/cfgtoollogs/opatch/opatch2010-04-30_12-51-17PM.log

Patch history file: /u01/app/grid/product/11.2.0/crs/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9352237  9343627  

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/grid/product/11.2.0/crs')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Execution of 'sh /u01/software/9352237/custom/scripts/pre -apply 9352237 ':

Return Code = 0

Applying patch 9352237...

ApplySession applying interim patch '9352237' to OH '/u01/app/grid/product/11.2.0/crs'
ApplySession: Optional component(s) [ oracle.oraolap, 11.2.0.1.0 ]  not present in the Oracle Home or a higher version is found.
Backing up files affected by the patch '9352237' for rollback. This might take a while...

Patching component oracle.rdbms.rsf, 11.2.0.1.0...
...
ApplySession adding interim patch '9343627' to inventory

Verifying the update...
Inventory check OK: Patch ID 9343627 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9343627 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
Running make for target ioracle
--------------------------------------------------------------------------------
********************************************************************************
********************************************************************************
**                                ATTENTION                                   **
**                                                                            **
** Please note that the Patch Set Update Installation (PSU Deinstallation)    **
** is not complete until all the Post Installation (Post Deinstallation)      **
** instructions noted in the Readme accompanying this PSU, have been          **
** successfully completed.                                                    **
**                                                                            **
********************************************************************************
********************************************************************************
--------------------------------------------------------------------------------
Execution of 'sh /u01/software/9352237/custom/scripts/post -apply 9352237 ':

Return Code = 0
UtilSession: N-Apply done.

OPatch succeeded.

Success! Now let’s move on to the RDBMS home. The readme suggests to roll-back any conflicting one-off patch from the RDBMS home. I didn’t apply any so don’t have to worry. Switching back to the Oracle account (owner of the RDBMS home), I applied the RDBMS part of the patch:

[oracle@rac11gr2node1 software]$ opatch napply 9343627/custom/server/ -local -oh $ORACLE_HOME -id 9343627
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
 from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2010-04-30_14-59-48PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9343627  

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11.2.0/dbhome_1')

Is the local system ready for patching? [y|n]
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...

Applying patch 9343627...

ApplySession applying interim patch '9343627' to OH '/u01/app/oracle/product/11.2.0/dbhome_1'
Backing up files affected by the patch '9343627' for rollback. This might take a while...
...
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/srvm/mesg/prvpus.msg"
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/srvm/mesg/sclcus.msg"
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/srvm/mesg/sclsus.msg"
ApplySession adding interim patch '9343627' to inventory

Verifying the update...
Inventory check OK: Patch ID 9343627 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9343627 are present in Oracle Home.
Running make for target install
Running make for target install

The local system has been patched and can be restarted.

UtilSession: N-Apply done.

OPatch succeeded.
[oracle@rac11gr2node1 software]$ opatch napply -local -oh $ORACLE_HOME -id 9352237
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
 from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2010-04-30_15-01-27PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9352237  

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks...
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: 

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11.2.0/dbhome_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...
Patching component oracle.rdbms.rsf, 11.2.0.1.0...
Updating archive file "/u01/app/oracle/product/11.2.0/dbhome_1/lib/libpls11.a"  with "lib/libpls11.a/phd.o"
Updating archive file "/u01/app/oracle/product/11.2.0/dbhome_1/lib/libpls11.a"  with "lib/libpls11.a/ph2drv.o"
...
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin/bundledata_PSU.xml"

Patching component oracle.oraolap, 11.2.0.1.0...
Updating archive file "/u01/app/oracle/product/11.2.0/dbhome_1/lib/liboraolap11.a"  with "lib/liboraolap11.a/xscarrot.o"

Patching component oracle.rdbms.deconfig, 11.2.0.1.0...
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/lib/libasmclnt11.so"

Patching component oracle.javavm.server, 11.2.0.1.0...
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/javavm/install/jvm_exp.sql"
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/javavm/install/initjvm.sql"
Copying file to "/u01/app/oracle/product/11.2.0/dbhome_1/javavm/install/jvmursc.sql"
ApplySession adding interim patch '9352237' to inventory

Verifying the update...
Inventory check OK: Patch ID 9352237 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9352237 are present in Oracle Home.
Running make for target client_sharedlib
Running make for target irman
Running make for target ikfod
Running make for target irenamedg
Running make for target idgmgrl
Running make for target imkpatch
Running make for target client_sharedlib
--------------------------------------------------------------------------------
********************************************************************************
********************************************************************************
**                                ATTENTION                                   **
**                                                                            **
** Please note that the Patch Set Update Installation (PSU Deinstallation)    **
** is not complete until all the Post Installation (Post Deinstallation)      **
** instructions noted in the Readme accompanying this PSU, have been          **
** successfully completed.                                                    **
**                                                                            **
********************************************************************************
********************************************************************************
--------------------------------------------------------------------------------

Execution of 'sh /u01/software/9352237/custom/scripts/post -apply 9352237 ':

Return Code = 0

The local system has been patched and can be restarted.

UtilSession: N-Apply done.

OPatch succeeded.
[oracle@rac11gr2node1 software]$

Post Patching

Wow, that took long but it seemed to have worked. What a massive patch! Now quickly do the post patch and clean up.

[oracle@rac11gr2node1 software]$ 9343627/custom/server/9343627/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME
Reading /u01/app/oracle/product/11.2.0/dbhome_1/install/params.ora..
Reading /u01/app/oracle/product/11.2.0/dbhome_1/install/params.ora..
Parsing file /u01/app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Parsing file /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Parsing file /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Parsing file /u01/app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Verifying file /u01/app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Verifying file /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Verifying file /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Verifying file /u01/app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/racgmain
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/racgeut
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/diskmon.bin
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/lsnodes
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/osdbagrp
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/rawutl
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/srvm/admin/ractrans
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/srvm/admin/getcrshome
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/gnsd
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/bin/crsdiag.pl
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libhasgen11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclsra11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libdbcfg11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libocr11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libocrb11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libocrutl11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libuini11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/librdjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libgns11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libgnsjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.2.0/dbhome_1/lib/libagfw11.so
[oracle@rac11gr2node1 software]$

Now, as root, re-lock the Grid Home:

[oracle@rac11gr2node1 software]$ su -
Password:
[root@rac11gr2node1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/grid/product/11.2.0/crs is /u01/app/oracle
[root@rac11gr2node1 ~]# $ORACLE_HOME/crs/install/rootcrs.pl -patch
2010-04-30 15:27:22: Parsing the host name
2010-04-30 15:27:22: Checking for super user privileges
2010-04-30 15:27:22: User has super user privileges
Using configuration parameter file: /u01/app/grid/product/11.2.0/crs/crs/install/crsconfig_params
CRS-4123: Oracle High Availability Services has been started.

Now start the grid home as the Grid Infrastructure owner:

[root@rac11gr2node1 ~]# su - grid
[grid@rac11gr2node1 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/grid/product/11.2.0/crs is /u01/app/oracle
[grid@rac11gr2node1 ~]$ srvctl start home -o $ORACLE_HOME -s /tmp/stat
stat       stat_grid  
[grid@rac11gr2node1 ~]$ srvctl start home -o $ORACLE_HOME -s /tmp/stat
stat       stat_grid  
[grid@rac11gr2node1 ~]$ srvctl start home -o $ORACLE_HOME -s /tmp/stat_grid -n rac11gr2node1
[grid@rac11gr2node1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Now start the RDBMS home and lean back for a second-you need to repeat this on all remaining cluster nodes now.

root@rac11gr2node1 ~]# su - oracle
[oracle@rac11gr2node1 ~]$ . oraenv
ORACLE_SID = [oracle] ? admindb
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is /u01/app/oracle
[oracle@rac11gr2node1 ~]$ srvctl start home -o $ORACLE_HOME -s /tmp/stat -n rac11gr2node1
[oracle@rac11gr2node1 ~]$

Eventually you will end up with a system that is patched! Once all nodes are patched, load modified SQL into the databases running out of the patched homes:

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

Check the logs for errors-I didn’t get any. You’re done-easily a Saturday….

11 Responses to “Applying PSU 11.2.0.1.1 to Grid Infrastructure & RDBMS”

  1. Dave Singleton said

    Dude, thanks!

    In pure “oracle” fashion, the psu readme was useless. you answered 3 questions based on oracle’s inability to speak english… as noted below.

    No mention of unlocking the home.
    The phrase “except for the local node if the GI home is shared”
    The use of the term administrative user, is at best ambiguous
    they suggest “opatch auto -oh ” vs. what you did…

    2.3.4 Patching the GI Home
    Before patching the GI home, make sure that the GI stack is shut down on all nodes in the cluster, except for the local node if the GI home is shared.

    To patch the GI home, as an administrative user execute a command in the following format:

    #opatch auto -oh
    You must run the preceding command in a rolling fashion on each node of the cluster.

    • rahul said

      how would i identify grid home is shared or not

      • Martin said

        Hi Rahul,

        a shared GRID_HOME is on a cluster file system of sorts. You’d check the output of the “mount” command for example-if it’s listing a CFS then you’re home is clustered/shared.

        HTH,

        Martin

  2. Dave Singleton said

    oh, and I’m applying, p9655006_112010_AIX64-5L.zip
    which came available 3 days ago?

    Wish me luck…

  3. […] Applying PSU 11.2.0.1.1 to grid infrastructure & RDBMS […]

  4. Martin,

    Have you got any action for the ones like me who overwrite the statfile :)

    • Martin said

      That’s nothing to worry about. All I can see is that the stats file is used to bring up the resources after the patch into the same state as they were before you shut down CRS for patching. So simply starting all the resources manually will do the trick :)

      BTW, good to see you today!

      • as you said there wasn’t anything to worry abou after I checked the content of the file :)

        Was pleasured to meet to today as well. Sorry that I could not say good bye (did not want to interrupt your conversation with Steve)

        Hope to come and catch you at Birmingham :)

  5. Monto said

    Martin,

    Thanks for the wonderful blog.I have to apply the PSU 2 on AIX 5L with RAC and ASM on 11201, can i follow the same steps for applying the patch?I think PSU 2 for GI also has PSU for rdbms too.Can you please guide on this as readme is very confusing for both patches(GI & RDBMS).

    Thanks

    • Martin said

      Hi Monto,

      I would assume you follow the same steps, but haven’t applied PSU 2 yet so take this with caution. I’d recommend raising a SR with Oracle before doing anything in this respect. Maybe we are all lucky and they rewrite the readme… One can still hope.

      Cheers,

      Martin

  6. Ryan said

    Great post. Just curious, after you applied the 11.2.0.1.1 patchset does your v$version table indicate 11.2.0.1.0 or 11.2.0.1.1?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: