A quick word of warning about database PSU 11.2.0.2.2

I am playing around with the Grid Infrastructure 11.2.0.2 PSU 2 and found an interesting note on My Oracle Support regarding the Patch Set Update. This reminds me that it’s always a good idea to search for a patch number on Metalink before applying a PSU. It also seems to be a good idea to wait for a few days before trying a PSU (or maybe CPU) on your DEV environment for the first time (and don’t even think about applying a PSU on production without thorough testing!)

OK, back to the story: there is a known issue with the patchset which has to do with the change in the Mutex behaviour which the PSU was intended to fix. To quote MOS note “Oracle Database Patch Set Update 11.2.0.2.2 Known Issues (Doc ID 1291879.1)”, Patch 12431716 Is a Recommended Patch for 11.2.0.2.2. In fact, Oracle strongly recommends you to apply the patch to fix Bug 12431716 – Unexpected change in mutex wait behavior in 11.2.0.2.2 PSU (higher CPU possible).

In a nutshell, not applying the patch can cause your system to suffer from excessive CPU usage and more than expected mutex contention. More information can be found in the description of Bug 12431716  Mutex waits may cause higher CPU usage in 11.2.0.2.2 PSU / GI PSU which is worth reading.

Besides this, the PSU was applied without any problems to my four node cluster, I just wish there was a way to roll out a new version of opatch to all cluster node’s $GRID_HOME and $ORACLE_HOME in one command. The overall process for the PSU is the same as already described in my previous post about Bundle Patch 3:

  1. Get the latest version of OPatch
  2. Deploy OPatch to $GRID_HOME and $ORACLE_HOME (ensure permissions are set correctly for the OPatch in $GRID_HOME!)
  3. Unzip the PSU (Bug 11724916 – 11.2.0.2.2 Patch Set Update (PSU) (Doc ID 11724916.8)), for example to /tmp/PSU
  4. Change directory to where you unzipped (/tmp/PSU) and become root
  5. Ensure that $GRID_HOME/OPatch is part of the path
  6. Read the readme
  7. Create an OCM response file and save it to say, /tmp/ocm.rsp
  8. Start the patch as root: opatch auto and supply the full path to the OCM response file (/tmp/ocm.rsp)
  9. Apply the beforementioned one-off patch

Then wait, and after a little while you spend trailing the logfile in $GRID_HOME/cfgtoollogs/ and having a coffee the process eventually finishes. Repeat on each node and you’re done. I’m really happy there aren’t these long readme files anymore with 8 steps to be performed, partially as root, partially as CRS owner/RDBMS owner. It reduces tge tune ut takes to apply a PSU significantly.

Happy patching!

Advertisements

22 thoughts on “A quick word of warning about database PSU 11.2.0.2.2”

  1. Vishal Gupta

    Martin, you could rollout the opatch to both $GRID_HOME and $RDBMS_HOME on all nodes easily.

    ( cd $GRID_HOME; unzip -o p6880880_112000_LInux-x86-64.zip; cd $RDBMS_HOME; unzip -o p6880880_112000_LInux-x86-64.zip)

    for i in `$GRID_HOME/bin/olsnodes |grep -v \`hostname -s\` `
    do
    scp -pr $GRID_HOME/OPatch $i:$GRID_HOME
    scp -pr $RDBMS_HOME/OPatch $i:$RDBMS_HOME
    done

    1. Martin Post author

      Hi Vishal!

      Great point, thank you. I’m in a situation where standards require me to keep the OPatch home as OPatch.old before installing the new version. Since $GRID_HOME is protected (owned by root) it’ll require coding around this. However on my current site I can’t “sudo” to unzip the patch in $GRID_HOME due to security concerns.

      Regards,

      Martin

  2. Martin Decker

    Hi Martin,

    indeed the regression of PSU 11.2.0.2.2 regarding Mutexes is a problem. However, ignoring PSU April is dangerous as well:
    ORA-600 or Data Corruption possible during shutdown normal/transactional/immediate of RAC instances in a rolling fashion [ID 1318986.1]

    Best regards,
    Martin

    1. Martin Post author

      Hi Martin,

      sorry if my message has been misleading, I definitely didn’t want to say that you shouldn’t apply the PSU. Contrary, I actually fully promote the PSU, I just wanted to make people aware that there is another one-off to be applied.

      Hope that helps,

      Martin

  3. coskan

    Great sum up Martin,

    I think I was wrong on our last discussion :) We did not apply PSU2 we applied the bundle.

    Thanks for the heads up

  4. Pingback: Log Buffer #221, A Carnival of the Vanities for DBAs | The Pythian Blog

  5. Wissem

    Hello Martin,

    Reading the Patch 12311357 – 11.2.0.2.2 GI Patch Set Update (Includes Database PSU 11.2.0.2.2) readme document.

    I don’t understand why should I unzip the GI PSU 11.2.0.2.2 patch in a shared location since I have installed the cluster in different Grid Infrastructure homes.

    Is it ok to unzip the patch in each cluster node and update the nodes separately?

    Extract of the read me file:

    “The patch application requires explicit user actions to run ‘opatch auto’ command on each node of Oracle clusterware. So, it is recommended that you download and unzip the GI PSU 11.2.0.2.2 patch in a shared location to be able to access it from any node in the cluster and then as the Grid home owner execute the unzip command.”

    Thank you

    1. Martin Bach Post author

      Good evening,

      it is of course ok to unzip the patch on each node-no trouble with that. The reason I chose a shared location (nfs mount) is convenience-just mount it once on the node, then execute the opatch auto and you are done. Saves space on the local file system as well.

      Best regards,

      Martin

  6. Wissem

    Good Evening Martin,

    Thank you for your help.
    Your RAC 11g book is great! I like it, keep writing :)

    Cheers,

    1. Martin Bach Post author

      Thanks for your comments, those kind of things make it all worthwhile!

      1. Wissem

        Hello Martin,

        I have applied the PSU Patch 12311357 – 11.2.0.2.2 GI Patch Set Update (Includes Database PSU 11.2.0.2.2).
        I have this entry in the registry$history table:
        SQL> select VERSION, COMMENTS from registry$history;

        VERSION COMMENTS
        ——————————————————————————–
        11.2.0.2 Patchset 11.2.0.2.0

        11.2.0.2 PSU 11.2.0.2.2

        SQL>

        After that, I applied the mutex patch 12431716 on only grid infrastructure home, is it OK to do it ONLY on grid infrastructure home OR I have to apply it also to the Oracle database homes ?

        Thank you

      2. Martin Bach Post author

        Hi!

        Just had a cursory glance at the readme of the patch. Interestingly, it doesn’t say which homes the patch should be applied for. I haven’t thought about it but my first reaction to your question was: apply everywhere. It addresses a problem with mutexes on the RDBMS instances, so at the very least apply the overlay patch on the RDBMS homes. If you are unsure, it’s always a good idea to raise a SR and discuss this with support-mention a documentation bug :)

        Hope this helps,

        Martin

      3. Wissem

        Thank you Martin.

        I am trying to create a listener on separate private bonding network:

        I did the following;

        ./srvctl add vip -n rac01 -A 192.168.71.55/255.255.255.0/bond2 -k 2

        but, when trying to start it, I have :

        /srvctl start vip -i rac01-om-vip
        PRCR-1079 : Failed to start resource ora.rac01-om-vip.vip
        CRS-2674: Start of ‘ora.net2.network’ on ‘rac01’ failed
        CRS-2674: Start of ‘ora.net2.network’ on ‘rac03’ failed
        CRS-2674: Start of ‘ora.net2.network’ on ‘rac02’ failed
        CRS-2674: Start of ‘ora.net2.network’ on ‘rac04’ failed
        CRS-2632: There are no more servers to try to place resource ‘ora.rac01-om-vip.vip’ on that would satisfy its placement policy

        Any idea?

      4. Martin Bach Post author

        Haven’t tested that myself, but I’m pretty sure that VIPs need to be started as root (ordinary users cannot start network resources). Have you tried that?

      6. Vishal Gupta

        You could check the applicability of a particular patch by followoing comand.

        $ORACLE_HOME/OPatch/opatch prereq CheckApplicable -oh $ORACLE_HOME -ph

    1. Martin Bach Post author

      Hi Younes,

      I don’t think you can, sorry.

      Best regards,

      Martin

  8. Wissem

    Hi Martin,
    How are you? Hope you are doing fine!

    I have a quick question, I am trying to apply the Patch 12311357 on my standalone grid infrastructure and single instance.

    But the thing is when trying to apply the PSU, it says:

    Cann’t open /a01/app/oracle/product/11.2.
    0/db_1/css/admin/init.cssd at /a01/app/oracle/product/11.2.0/db_1/OPatch/crs/auto_patch.pl line 3161.

    It seems like the PSU considers that is a cluster database. In all the readme document it doesn’t mention single instance database.

    3 weeks ago, I successfully applied it in RAC 4 nodes, without issues.

    I wonder if you have tried it on a single instance?

    Thank you Martin,
    Wissem

    1. Martin Bach Post author

      Hi Wissem,

      I haven’t tried this I’m afraid. I haven’t checked, but is there a different patch ID for single instance?

      Martin

      1. Wissem

        Hi Martin,

        I don’t see any different patch ID.
        I will look into it again, and I will let you know,

        Thank you,
        Wissem

Comments are closed.