When I started the series about Tracefile Analyzer (TFA) I promised three parts. One for single instance, another one for Oracle Restart and this one is going to be about Real Application Clusters. The previous two parts are published already, this is the final piece.
The environment
I am using a virtualised 2-node cluster named rac122pri with nodes rac122pri1{1,2} based on Oracle Linux 7.4. RAC is patched to 12.2.0.1.180116. I installed a Grid Home and an RDBMS home (Enterprise Edition).
Real Application Clusters
Before starting this discussion it’s worth pointing out that TFA integration in RAC 12.2 works really well. TFA is installed as part of the initial setup of the binaries and documented in the Autonomous Health Framework.
As soon as you execute root.sh as part of your RAC 12.2 Grid Infrastructure installation, TFA is set up as the first step out of 19. Unfortunately I don’t have the screen output available, but all the action is logged in rootcrs_$(hostname)_timestamp.log anyway. You can find that logfile in $ORACLE_BASE/crsdata/$(hostname)/crsconfig. If memory serves me right this is a 12.2 change, I think the file was in $GRID_HOME/cfgtoollogs in 12.1. but I’m not sure (and don’t have a system to check against).
Here is the relevant output from the log file:
2018-05-11 12:16:02: Command output: > CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'. >End Command output 2018-05-11 12:16:02: CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'. 2018-05-11 12:16:02: Executing cmd: /u01/app/12.2.0.1/grid/bin/clsecho -p has -f clsrsc -m 4001 2018-05-11 12:16:02: Command output: > CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector. >End Command output 2018-05-11 12:16:02: CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector. 2018-05-11 12:16:02: Executing cmd: /u01/app/12.2.0.1/grid/crs/install/tfa_setup -silent -crshome /u01/app/12.2.0.1/grid 2018-05-11 12:16:24: Command output: > TFA Installation Log will be written to File : /tmp/tfa_install_8879_2018_05_11-12_16_02.log > > Starting TFA installation > > Using JAVA_HOME : /u01/app/12.2.0.1/grid/jdk/jre > > Running Auto Setup for TFA as user root... > > Installing TFA now... > > > TFA Will be Installed on rac122pri1... > > TFA will scan the following Directories > ++++++++++++++++++++++++++++++++++++++++++++ > > .---------------------------------------------------------. > | rac122pri1 | > +----------------------------------------------+----------+ > | Trace Directory | Resource | > +----------------------------------------------+----------+ > | /u01/app/12.2.0.1/grid/OPatch/crs/log | CRS | ... [a lot more directories] > | /u01/app/oracle/crsdata/rac122pri1/trace | CRS | > '----------------------------------------------+----------' > > > Installing TFA on rac122pri1: > HOST: rac122pri1 TFA_HOME: /u01/app/12.2.0.1/grid/tfa/rac122pri1/tfa_home > > .------------------------------------------------------------------------------. > | Host | Status of TFA | PID | Port | Version | Build ID | > +------------+---------------+------+------+------------+----------------------+ > | rac122pri1 | RUNNING | 9165 | 5000 | 12.2.1.0.0 | 12210020161122170355 | > '------------+---------------+------+------+------------+----------------------' > > Running Inventory in All Nodes... > > Enabling Access for Non-root Users on rac122pri1... > > Adding default users to TFA Access list... > > Summary of TFA Installation: > .----------------------------------------------------------------------. > | rac122pri1 | > +---------------------+------------------------------------------------+ > | Parameter | Value | > +---------------------+------------------------------------------------+ > | Install location | /u01/app/12.2.0.1/grid/tfa/rac122pri1/tfa_home | > | Repository location | /u01/app/oracle/tfa/repository | > | Repository usage | 0 MB out of 10240 MB | > '---------------------+------------------------------------------------' > > > Installing oratop extension.. > > > TFA is successfully installed... ... >End Command output 2018-05-11 12:16:24: Executing cmd: /u01/app/12.2.0.1/grid/bin/clsecho -p has -f clsrsc -m 4002 2018-05-11 12:16:24: Command output: > CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector. >End Command output
As with the other database deployment types, TFA integrates into systemd on Oracle Linux 7:
[root@rac122pri1 ~]# systemctl status oracle-tfa
● oracle-tfa.service - Oracle Trace File Analyzer
Loaded: loaded (/etc/systemd/system/oracle-tfa.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-06-20 23:03:11 BST; 15h ago
Main PID: 3092 (init.tfa)
CGroup: /system.slice/oracle-tfa.service
├─ 3092 /bin/sh /etc/init.d/init.tfa run >/dev/null 2>&1 </dev/null
├─ 3698 /u01/app/12.2.0.1/grid/tfa/rac122pri1/tfa_home/jre/bin/java -
server -Xms32m -Xmx64m -Djava.awt.headless=true -Ddisable.checkForUpdate=true
oracle.rat.tfa.TFAMain /u01/app/12.2.0.1/grid/tfa/rac122pri1/...
└─15827 /bin/sleep 30
The service is enabled and starts together with the operating system.
RAC-TFA lives in the Grid Home
If you read the previous parts of this series you might have noticed in the previous output that unlike single instance and Oracle Restart environments, TFA is resident in the Grid Home as opposed to ORACLE_BASE. This could have implications if you are using a system implementing separation of duties. For everyone else it’s a question of making sure the environment is set correctly.
[oracle@rac122pri1 ~]$ . oraenv ORACLE_SID = [oracle] ? +ASM1 The Oracle base has been set to /u01/app/oracle [oracle@rac122pri1 ~]$ which tfactl /u01/app/12.2.0.1/grid/bin/tfactl
Another difference between single node and RAC installations is the status message. TFA reports its status on all nodes in the cluster, as shown here:
[oracle@rac122pri1 ~]$ tfactl print status .-------------------------------------------------------------------------------------------------. | Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status | +------------+---------------+------+------+------------+----------------------+------------------+ | rac122pri1 | RUNNING | 3698 | 5000 | 18.2.1.0.0 | 18210020180529111033 | COMPLETE | | rac122pri2 | RUNNING | 3697 | 5000 | 18.2.1.0.0 | 18210020180529111033 | COMPLETE | '------------+---------------+------+------+------------+----------------------+------------------' [oracle@rac122pri1 ~]$
This isn’t the stock version by the way, I have recently upgraded my deployment to 18.2.1.0. More about that later. I recently installed RAC 12.2 and when invoking TFA am greeted with the familiar “WARNING – your software is older than 180 days. Please consider upgrading TFA to the latest version”.
You may want to consider upgrading the installation
As with the other deployments, the version you get with the base installation is most likely outdated. My base release showed this:
.-------------------------------------------------------------------------------------------------. | Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status | +------------+---------------+------+------+------------+----------------------+------------------+ | rac122pri1 | RUNNING | 3698 | 5000 | 12.2.1.0.0 | 12210020161122170355 | COMPLETE | | rac122pri2 | RUNNING | 3697 | 5000 | 12.2.1.0.0 | 12210020161122170355 | COMPLETE | '------------+---------------+------+------+------------+----------------------+------------------'
But not only that: if you check for toolstatus you will notice that most of the support tools are missing just like with the other deployments. I love to have them and as such upgraded to the current version at the time of writing (18.2.1.0.0). My Oracle Support DOC ID 1513912.1 is the reference, as always.
After the upgrade completed, I could see the following tools:
[oracle@rac122pri1 ~]$ tfactl toolstatus .------------------------------------------------------------------. | TOOLS STATUS - HOST : rac122pri1 | +----------------------+--------------+--------------+-------------+ | Tool Type | Tool | Version | Status | +----------------------+--------------+--------------+-------------+ | Development Tools | orachk | 12.2.0.1.3 | DEPLOYED | | | oratop | 14.1.2 | DEPLOYED | +----------------------+--------------+--------------+-------------+ | Support Tools Bundle | darda | 2.10.0.R6036 | DEPLOYED | | | oswbb | 8.0.1 | RUNNING | | | prw | 12.1.13.11.4 | NOT RUNNING | +----------------------+--------------+--------------+-------------+ | TFA Utilities | alertsummary | 12.2.1.1.0 | DEPLOYED | | | calog | 12.2.0.1.0 | DEPLOYED | | | changes | 12.2.1.1.0 | DEPLOYED | | | dbglevel | 12.2.1.1.0 | DEPLOYED | | | events | 12.2.1.1.0 | DEPLOYED | | | grep | 12.2.1.1.0 | DEPLOYED | | | history | 12.2.1.1.0 | DEPLOYED | | | ls | 12.2.1.1.0 | DEPLOYED | | | managelogs | 12.2.1.1.0 | DEPLOYED | | | menu | 12.2.1.1.0 | DEPLOYED | | | param | 12.2.1.1.0 | DEPLOYED | | | ps | 12.2.1.1.0 | DEPLOYED | | | pstack | 12.2.1.1.0 | DEPLOYED | | | search | 18.2.0.0.0 | DEPLOYED | | | summary | 12.2.1.1.0 | DEPLOYED | | | tail | 12.2.1.1.0 | DEPLOYED | | | triage | 12.2.1.1.0 | DEPLOYED | | | vi | 12.2.1.1.0 | DEPLOYED | '----------------------+--------------+--------------+-------------' Note :- DEPLOYED : Installed and Available - To be configured or run interactively. NOT RUNNING : Configured and Available - Currently turned off interactively. RUNNING : Configured and Available.
Summary
Whilst TFA is a great tool for single instance and Oracle Restart environments, it is indispensible for RAC. The ability to gather a lot of diagnostic data that would otherwise be difficult to get is the prime use case. Tools such as OSWatcher are very useful when it comes to diagnosing system status at the time of an instance eviction for example.
