I am playing around with the Grid Infrastructure 11.2.0.2 PSU 2 and found an interesting note on My Oracle Support regarding the Patch Set Update. This reminds me that it’s always a good idea to search for a patch number on Metalink before applying a PSU. It also seems to be a good idea to wait for a few days before trying a PSU (or maybe CPU) on your DEV environment for the first time (and don’t even think about applying a PSU on production without thorough testing!)
OK, back to the story: there is a known issue with the patchset which has to do with the change in the Mutex behaviour which the PSU was intended to fix. To quote MOS note “Oracle Database Patch Set Update 11.2.0.2.2 Known Issues (Doc ID 1291879.1)”, Patch 12431716 Is a Recommended Patch for 11.2.0.2.2. In fact, Oracle strongly recommends you to apply the patch to fix Bug 12431716 – Unexpected change in mutex wait behavior in 11.2.0.2.2 PSU (higher CPU possible).
In a nutshell, not applying the patch can cause your system to suffer from excessive CPU usage and more than expected mutex contention. More information can be found in the description of Bug 12431716 Mutex waits may cause higher CPU usage in 11.2.0.2.2 PSU / GI PSU which is worth reading.
Besides this, the PSU was applied without any problems to my four node cluster, I just wish there was a way to roll out a new version of opatch to all cluster node’s $GRID_HOME and $ORACLE_HOME in one command. The overall process for the PSU is the same as already described in my previous post about Bundle Patch 3:
- Get the latest version of OPatch
- Deploy OPatch to $GRID_HOME and $ORACLE_HOME (ensure permissions are set correctly for the OPatch in $GRID_HOME!)
- Unzip the PSU (Bug 11724916 – 11.2.0.2.2 Patch Set Update (PSU) (Doc ID 11724916.8)), for example to /tmp/PSU
- Change directory to where you unzipped (/tmp/PSU) and become root
- Ensure that $GRID_HOME/OPatch is part of the path
- Read the readme
- Create an OCM response file and save it to say, /tmp/ocm.rsp
- Start the patch as root: opatch auto and supply the full path to the OCM response file (/tmp/ocm.rsp)
- Apply the beforementioned one-off patch
Then wait, and after a little while you spend trailing the logfile in $GRID_HOME/cfgtoollogs/ and having a coffee the process eventually finishes. Repeat on each node and you’re done. I’m really happy there aren’t these long readme files anymore with 8 steps to be performed, partially as root, partially as CRS owner/RDBMS owner. It reduces tge tune ut takes to apply a PSU significantly.
Happy patching!
Martin, you could rollout the opatch to both $GRID_HOME and $RDBMS_HOME on all nodes easily.
( cd $GRID_HOME; unzip -o p6880880_112000_LInux-x86-64.zip; cd $RDBMS_HOME; unzip -o p6880880_112000_LInux-x86-64.zip)
for i in `$GRID_HOME/bin/olsnodes |grep -v \`hostname -s\` `
do
scp -pr $GRID_HOME/OPatch $i:$GRID_HOME
scp -pr $RDBMS_HOME/OPatch $i:$RDBMS_HOME
done
Hi Vishal!
Great point, thank you. I’m in a situation where standards require me to keep the OPatch home as OPatch.old before installing the new version. Since $GRID_HOME is protected (owned by root) it’ll require coding around this. However on my current site I can’t “sudo” to unzip the patch in $GRID_HOME due to security concerns.
Regards,
Martin
Hi Martin,
indeed the regression of PSU 11.2.0.2.2 regarding Mutexes is a problem. However, ignoring PSU April is dangerous as well:
ORA-600 or Data Corruption possible during shutdown normal/transactional/immediate of RAC instances in a rolling fashion [ID 1318986.1]
Best regards,
Martin
Hi Martin,
sorry if my message has been misleading, I definitely didn’t want to say that you shouldn’t apply the PSU. Contrary, I actually fully promote the PSU, I just wanted to make people aware that there is another one-off to be applied.
Hope that helps,
Martin
Great sum up Martin,
I think I was wrong on our last discussion :) We did not apply PSU2 we applied the bundle.
Thanks for the heads up
Pingback: Log Buffer #221, A Carnival of the Vanities for DBAs | The Pythian Blog
Hello Martin,
Reading the Patch 12311357 – 11.2.0.2.2 GI Patch Set Update (Includes Database PSU 11.2.0.2.2) readme document.
I don’t understand why should I unzip the GI PSU 11.2.0.2.2 patch in a shared location since I have installed the cluster in different Grid Infrastructure homes.
Is it ok to unzip the patch in each cluster node and update the nodes separately?
Extract of the read me file:
“The patch application requires explicit user actions to run ‘opatch auto’ command on each node of Oracle clusterware. So, it is recommended that you download and unzip the GI PSU 11.2.0.2.2 patch in a shared location to be able to access it from any node in the cluster and then as the Grid home owner execute the unzip command.”
Thank you
Good evening,
it is of course ok to unzip the patch on each node-no trouble with that. The reason I chose a shared location (nfs mount) is convenience-just mount it once on the node, then execute the opatch auto and you are done. Saves space on the local file system as well.
Best regards,
Martin
Good Evening Martin,
Thank you for your help.
Your RAC 11g book is great! I like it, keep writing :)
Cheers,
Thanks for your comments, those kind of things make it all worthwhile!
Hello Martin,
I have applied the PSU Patch 12311357 – 11.2.0.2.2 GI Patch Set Update (Includes Database PSU 11.2.0.2.2).
I have this entry in the registry$history table:
SQL> select VERSION, COMMENTS from registry$history;
VERSION COMMENTS
——————————————————————————–
11.2.0.2 Patchset 11.2.0.2.0
11.2.0.2 PSU 11.2.0.2.2
SQL>
After that, I applied the mutex patch 12431716 on only grid infrastructure home, is it OK to do it ONLY on grid infrastructure home OR I have to apply it also to the Oracle database homes ?
Thank you
Hi!
Just had a cursory glance at the readme of the patch. Interestingly, it doesn’t say which homes the patch should be applied for. I haven’t thought about it but my first reaction to your question was: apply everywhere. It addresses a problem with mutexes on the RDBMS instances, so at the very least apply the overlay patch on the RDBMS homes. If you are unsure, it’s always a good idea to raise a SR and discuss this with support-mention a documentation bug :)
Hope this helps,
Martin
Thank you Martin.
I am trying to create a listener on separate private bonding network:
I did the following;
./srvctl add vip -n rac01 -A 192.168.71.55/255.255.255.0/bond2 -k 2
but, when trying to start it, I have :
/srvctl start vip -i rac01-om-vip
PRCR-1079 : Failed to start resource ora.rac01-om-vip.vip
CRS-2674: Start of ‘ora.net2.network’ on ‘rac01’ failed
CRS-2674: Start of ‘ora.net2.network’ on ‘rac03’ failed
CRS-2674: Start of ‘ora.net2.network’ on ‘rac02’ failed
CRS-2674: Start of ‘ora.net2.network’ on ‘rac04’ failed
CRS-2632: There are no more servers to try to place resource ‘ora.rac01-om-vip.vip’ on that would satisfy its placement policy
Any idea?
Haven’t tested that myself, but I’m pretty sure that VIPs need to be started as root (ordinary users cannot start network resources). Have you tried that?
Thank you Martin , I am going to do it now, and let you know :)
You could check the applicability of a particular patch by followoing comand.
$ORACLE_HOME/OPatch/opatch prereq CheckApplicable -oh $ORACLE_HOME -ph
Hi Martin,
Can I use opatch auto for 12431716?
Thanks
Hi Younes,
I don’t think you can, sorry.
Best regards,
Martin
Hi Martin,
How are you? Hope you are doing fine!
I have a quick question, I am trying to apply the Patch 12311357 on my standalone grid infrastructure and single instance.
But the thing is when trying to apply the PSU, it says:
Cann’t open /a01/app/oracle/product/11.2.
0/db_1/css/admin/init.cssd at /a01/app/oracle/product/11.2.0/db_1/OPatch/crs/auto_patch.pl line 3161.
It seems like the PSU considers that is a cluster database. In all the readme document it doesn’t mention single instance database.
3 weeks ago, I successfully applied it in RAC 4 nodes, without issues.
I wonder if you have tried it on a single instance?
Thank you Martin,
Wissem
Hi Wissem,
I haven’t tried this I’m afraid. I haven’t checked, but is there a different patch ID for single instance?
Martin
Hi Martin,
I don’t see any different patch ID.
I will look into it again, and I will let you know,
Thank you,
Wissem
Hi Martin,
I have successfully applied the patch, here my experience http://wp.me/p1hlba-xD
thanks :)
Wissem