I have previously written about changes in the Oracle 12.2 preinstall RPM and how gcc is no longer part of the dependencies list. As was pointed out to me, this shouldn’t be necessary anymore, according to the 12.2 Linux Database Installation Guide. Check the blue note for a statement indicating that gcc and gcc-c++ aren’t needed for Grid Infrastructure, nor for the RDBMS software.
I have applied patch 27100009 (January 2018 Release Update 220.127.116.11.180116) on my 2 node RAC system in the lab, and found out that this is partially true :) You may or may not encounter this issue in your environment, see below.
Updating the preinstall RPM
Just to be sure I didn’t miss any changes to the preinstall RPM I pulled the latest one from Oracle and checked it’s requirements for gcc.
[oracle@rac122pri1 ~]$ rpm -qi oracle-database-server-12cR2-preinstall-1.0-3.el7.x86_64 Name : oracle-database-server-12cR2-preinstall Version : 1.0 Release : 3.el7 Architecture: x86_64 Install Date: Fri 19 Jan 2018 03:13:18 PM GMT Group : Test Environment/Libraries Size : 56561 License : GPLv2 Signature : RSA/SHA256, Mon 10 Jul 2017 11:27:07 AM BST, Key ID 72f97b74ec551f03 Source RPM : oracle-database-server-12cR2-preinstall-1.0-3.el7.src.rpm Build Date : Mon 10 Jul 2017 11:26:59 AM BST Build Host : x86-ol7-builder-02.us.oracle.com Relocations : (not relocatable) Vendor : Oracle Summary : Sets the system for Oracle Database single instance and Real Application Cluster install for Oracle Linux 7 Description : The Oracle Preinstallation RPM package installs software packages and sets system parameters required for Oracle Database single instance and Oracle Real Application Clusters installations for Oracle Linux Release 7 Files affected: /etc/sysctl.conf, /boot/grub/menu.lst OR /boot/grub2/grub.cfg Files added: /etc/security/limits.d/oracle-database-server-12cR2-preinstall.conf [oracle@rac122pri1 ~]$ rpm -q --requires oracle-database-server-12cR2-preinstall-1.0-3.el7.x86_64 | grep -i gcc libgcc [oracle@rac122pri1 ~]$
So gcc is definitely not part of the dependent packages, and my minimum install doesn’t pull gcc (or gcc-c++ for that matter):
[oracle@rac122pri1 ~]$ rpm -qa | grep gcc libgcc-4.8.5-16.el7_4.1.x86_64 [oracle@rac122pri1 ~]$
This is a 2 node cluster, consisting of nodes rac122pri1 and rac122pri2. Both use Oracle Linux 7.4 with UEK4 patched to February 12th 2018. As the names I picked for my cluster nodes suggest, this is a system running RAC 12.2.
IMPORTANT note: just because I hit the issue doesn’t mean there’s an issue for everyone. My lab environment is that – my lab environment. I just wanted to point out a way to investigate the problem and how I resolved it in my lab environment. Your mileage might vary.
Applying the January 2018 RU
I always like to be current when it comes to Oracle, and in that spirit decided to apply the January 2018 RU to my cluster after having made sure that I have a working backup. My rule is that there is no patching without backups, ever. And by backups I mean backups of the entire stack ;)
Since this was a new installation in my lab I thought I’d give opatchauto another chance and run with it. What could possibly go wrong?
Node 1 went without any issues, and opatchauto reported that it had applied the patches it wanted to apply. I don’t have the screen output anymore, however here’s the summary after the patch completed.
[oracle@rac122pri1 ~]$ opatch lspatches -oh /u01/app/oracle/product/18.104.22.168/dbhome_1 27335416;OCW JAN 2018 RELEASE UPDATE 22.214.171.124.180116 (27335416) 27105253;Database Release Update : 126.96.36.199.180116 (27105253) OPatch succeeded. [oracle@rac122pri1 ~]$ opatch lspatches -oh /u01/app/188.8.131.52/grid/ 27335416;OCW JAN 2018 RELEASE UPDATE 184.108.40.206.180116 (27335416) 27144050;Tomcat Release Update 220.127.116.11.0(ID:171023.0830) (27144050) 27128906;ACFS Release Update : 18.104.22.168.0 (27128906) 27105253;Database Release Update : 22.214.171.124.180116 (27105253) 26839277;DBWLM RELEASE UPDATE 126.96.36.199.0(ID:170913) (26839277) OPatch succeeded. [oracle@rac122pri1 ~]$
Did you notice the “Tomcat Release Update 188.8.131.52.0” in the Grid Home? So much to investigate!
Patching node 2
After the patch completed on node 1 I continued with node 2. At first, everything looked quite normal, and opatch did it’s thing. I have taken a habit to always apply anything that depends on the network connection to be up in a screen (1) session. That protects the work I’m doing from intermittent network trouble, and allows me to keep a view on how things are progressing.
After a few minutes of work, opatchauto reported this (formatted for readability):
Start applying binary patch on home /u01/app/184.108.40.206/grid Failed while applying binary patches on home /u01/app/220.127.116.11/grid Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures: Patch Target : rac122pri2->/u01/app/18.104.22.168/grid Type[crs] Details: [ ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /u01/app/22.214.171.124/grid, host: rac122pri2. Command failed: /u01/app/126.96.36.199/grid/OPatch/opatchauto apply /u01/stage/27100009 -oh /u01/app/188.8.131.52/grid -target_type cluster -binary -invPtrLoc /u01/app/184.108.40.206/grid/oraInst.loc -jre /u01/app/220.127.116.11/grid/OPatch/jre -persistresult /u01/app/18.104.22.168/grid/OPatch/auto/dbsessioninfo/sessionresult_rac122pri2_crs.ser -analyzedresult /u01/app/22.214.171.124/grid/OPatch/auto/dbsessioninfo/sessionresult_analyze_rac122pri2_crs.ser Command failure output: ==Following patches FAILED in apply: Patch: /u01/stage/27100009/27335416 Log: /u01/app/126.96.36.199/grid/cfgtoollogs/opatchauto/core/opatch/opatch2018-01-22_09-37-57AM_1.log Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "install_srvm".
Not exactly what you want to see during patching. But there’s no need to panic (just yet ;) The error message points me to a log file. Opening the log file it quickly become apparent why the issue occurred:
[22-Jan-2018 09:40:30] [INFO] [OPSR-TIME] Finished applying patch "27335416" to local system [22-Jan-2018 09:40:31] [INFO] [OPSR-TIME] Loading raw inventory [22-Jan-2018 09:40:31] [INFO] [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 314 (MB) [22-Jan-2018 09:40:31] [INFO] [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 314 (MB) [22-Jan-2018 09:40:31] [INFO] [OPSR-TIME] Raw inventory loaded successfully [22-Jan-2018 09:40:31] [INFO] [OPSR-TIME] Loading cooked inventory [22-Jan-2018 09:40:31] [INFO] [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 314 (MB) [22-Jan-2018 09:40:32] [INFO] [OPSR-MEMORY] : Loaded cooked oneoffs. Heap memory used : 363 (MB) [22-Jan-2018 09:40:32] [INFO] [OPSR-TIME] Cooked inventory loaded successfully [22-Jan-2018 09:40:32] [INFO] OUI-67050:Running make for target install_srvm [22-Jan-2018 09:40:32] [INFO] Start invoking 'make' at Mon Jan 22 09:40:32 GMT 2018Mon Jan 22 09:40:32 GMT 2018 [22-Jan-2018 09:40:32] [INFO] Finish invoking 'make' at Mon Jan 22 09:40:32 GMT 2018 [22-Jan-2018 09:40:32] [WARNING] OUI-67200:Make failed to invoke "/usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=/u01/app/188.8.131.52/grid"....'/bin/sh: /usr/bin/gcc: No such file or directory make: *** [/u01/app/184.108.40.206/grid/rdbms/lib/config.o] Error 127 ' [22-Jan-2018 09:40:32] [INFO] Stack Description: java.lang.RuntimeException: /bin/sh: /usr/bin/gcc: No such file or directory make: *** [/u01/app/220.127.116.11/grid/rdbms/lib/config.o] Error 127 [22-Jan-2018 09:40:32] [INFO] StackTrace: oracle.opatch.MakeAction.apply(MakeAction.java:534)
This is actually quite simple: for some reason opatch wants to use gcc, and doesn’t find it. After installing gcc and dependencies, I resumed opatchauto and finished patching successfully. I haven’t understood why Oracle wants to use gcc on node 2 after it didn’t require it on node 1.
Just thought I’d pass this on in case you hit the same problem. Happy patching!