OSWatcher integration in Trace File Analyzer (TFA)

Some time ago I wrote a post about using OSWatcher for system analysis. Neil Chandler (@ChandlerDBA) rightfully pointed out that although OSWatcher on its own was all right, TFA was the way to go. TFA can include OSWatcher, but more importantly it adds a lot of value over and above what OSWatcher does.

Update

Since I published this post in 2016 a lot has happened, requiring a rewrite of the post in April 2020. The most important change for users is the merge of TFA into the new Autonomous Health Framework (AHF). You download AHF from My Oracle Support Doc ID 2550798.1.

Should I use OSWatcher standalone or as part of AHF?

I guess it depends on what you want to do-I still think that OSWatcher is a good starting point and enough for most problems on single instance systems provided you don’t forget to add OSWatcher to systemd to start when the system boots up.

When it comes to clustered environments or Oracle Restart, AHF looks a lot more appealing.

This post covers AHF 20.1.2 for Linux, the current version at the time of writing. The screen output has been generated on Oracle Linux 7.8 running in a KVM VM. A single-instance lab database is creating some noise.

What is TFA?

TFA (now part of AHF) is a tool which – among other things – helps you gather information about incidents. For clustered environment, it can do so across all the nodes. If you ever worked on Exadata half-racks or other clusters with more than 4 nodes you will quickly start to appreciate having to use one tool for this task. The TFA output is suitable for attaching to a Service Request which should, at least in theory, help speed up the problem resolution.

As an added benefit you get a lot of tools that were previously known as “RAC and DB Support Tools Bundle”. This includes OSWatcher as well, the reason for this post.

Running OSWatcher as part of TFA has one key benefit: you don’t have to worry about starting OSWatcher when booting. TFA is started via a systemd unit file in Oracle Linux 7. You can check its status using the standard systemd commands suite, as shown here:

[root@server5 ahf]# systemctl status oracle-tfa
● oracle-tfa.service - Oracle Trace File Analyzer
   Loaded: loaded (/etc/systemd/system/oracle-tfa.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-04-23 14:34:13 BST; 1min 17s ago
 Main PID: 11095 (init.tfa)
   CGroup: /system.slice/oracle-tfa.service
           ├─11095 /bin/sh /etc/init.d/init.tfa run >/dev/null 2>&1 

Support Tools Bundle

Earlier I said you benefit from the latest Support Tools Bundle when installing AHF. At the time of writing, this included the following tools:

[root@server5 ahf]# /opt/oracle.ahf/bin/tfactl status

.------------------------------------------------------------------------------------------------.
| Host    | Status of TFA | PID   | Port  | Version    | Build ID             | Inventory Status |
+---------+---------------+-------+-------+------------+----------------------+------------------+
| server5 | RUNNING       | 16762 | 25805 | 20.1.2.0.0 | 20120020200403113404 | COMPLETE         |
'---------+---------------+-------+-------+------------+----------------------+------------------'

[root@server5 ahf]# /opt/oracle.ahf/bin/tfactl toolstatus

.------------------------------------------------------------------.
|                   TOOLS STATUS - HOST : server5                  |
+----------------------+--------------+--------------+-------------+
| Tool Type            | Tool         | Version      | Status      |
+----------------------+--------------+--------------+-------------+
| Development Tools    | orachk       |   19.3.0.0.0 | DEPLOYED    |
|                      | oratop       |       14.1.2 | DEPLOYED    |
+----------------------+--------------+--------------+-------------+
| Support Tools Bundle | darda        | 2.10.0.R6036 | DEPLOYED    |
|                      | oswbb        |        8.3.2 | RUNNING     |
|                      | prw          | 12.1.13.11.4 | NOT RUNNING |
+----------------------+--------------+--------------+-------------+
| TFA Utilities        | alertsummary |   19.3.0.0.0 | DEPLOYED    |
|                      | calog        |   19.3.0.0.0 | DEPLOYED    |
|                      | dbcheck      |   18.3.0.0.0 | DEPLOYED    |
|                      | dbglevel     |   19.3.0.0.0 | DEPLOYED    |
|                      | grep         |   19.3.0.0.0 | DEPLOYED    |
|                      | history      |   19.3.0.0.0 | DEPLOYED    |
|                      | ls           |   19.3.0.0.0 | DEPLOYED    |
|                      | managelogs   |   19.3.0.0.0 | DEPLOYED    |
|                      | menu         |   19.3.0.0.0 | DEPLOYED    |
|                      | param        |   19.3.0.0.0 | DEPLOYED    |
|                      | ps           |   19.3.0.0.0 | DEPLOYED    |
|                      | pstack       |   19.3.0.0.0 | DEPLOYED    |
|                      | summary      |   19.3.0.0.0 | DEPLOYED    |
|                      | tail         |   19.3.0.0.0 | DEPLOYED    |
|                      | triage       |   19.3.0.0.0 | DEPLOYED    |
|                      | vi           |   19.3.0.0.0 | DEPLOYED    |
'----------------------+--------------+--------------+-------------'

Note :-
  DEPLOYED    : Installed and Available - To be configured or run interactively.
  NOT RUNNING : Configured and Available - Currently turned off interactively.
  RUNNING     : Configured and Available.

[root@server5 ahf]# 

Invoking OSwatcher

With OSWatcher running and gathering data, you can invoke it in the usual way:

[oracle@server5 ~]$ tfactl oswbb -h

Usage : /opt/oracle.ahf/tfa/bin/tfactl.pl [run] oswbb [ | -since n[mhd] ]

Options:

-since n[mhd] Run OSWatcher analyzer for last n [m]inutes or [h]ours or [d]ays.

: -P  -L  -6 -7 -8 -B  -E  -A
     -P   User specified name of the html profile generated
                        by oswbba. This overrides the oswbba automatic naming
                        convention for html profiles. All profiles
                        whether user specified named or auto generated
                        named will be located in the /profile directory.

     -A  Same as option A from the menu. Will generate
                        an analysis report in the /analysis directory or
                        user can also specify the name of the analysis file
                        by specifying full qualified path name of file.
                        The "A" option can not be used together with the
                        "S" option.
     -S <>              Will generate an analysis of a subset of the data
                        in the archive directory. This option must be used
                        together with the -b and -e options below. See the
                        section "Specifying the begin/end time of the analysis"
                        above. The "S" option can not be used together with
                        the "A" option.

     -START   Used with the analysis option to specify the first
                        file located in the oswvmstat directory to analyze.

     -STOP    Used with the analysis option to specify the last
                        file located in the oswvmstat directory to analyze.

     -b     Used with the -S option to specify the begin time
                        of the analysis period. Example format:
                        -b Jan 09 13:00:00 2013

     -e       Used with the -S option to specify the end time
                        of the analysis period. Example format:
                        -e Jan 09 13:15:00 2013

     -L  User specified location of an existing directory
                        to place any gif files generated
                        by oswbba. This overrides the oswbba automatic
                        convention for placing all gif files in the
                        /gif directory. This directory must pre-exist!
     -6                 Same as option 6 from the menu. Will generate
                        all cpu gif files.


     -7                 Same as option 7 from the menu. Will generate
                        all memory gif files.

     -8                 Same as option 8 from the menu. Will generate
                        all disk gif files.



     -NO_IOSTAT         Ignores files in the oswiostat directory from
                        analysis

     -NO_TOP            Ignores files in the oswtop directory from
                        analysis

     -NO_NETSTAT        Ignores files in the oswnetstat directory from
                        analysis

     -NO_PS             Ignores files in the oswps directory from
                        analysis

     -MEM_ALL           Analyzes virtual and resident memory allocations
                        for all processes. This is very resource intensive.

     -NO_Linux          Ignores files in the oswmeminfo directory from
                        analysis

e.g:
   /opt/oracle.ahf/tfa/bin/tfactl.pl oswbb
   /opt/oracle.ahf/tfa/bin/tfactl.pl oswbb -since 2h

   /opt/oracle.ahf/tfa/bin/tfactl.pl run oswbb
   /opt/oracle.ahf/tfa/bin/tfactl.pl run oswbb -since 2h 

Summary

TFA really is a very useful tool, and this is not only due to the integration of OSWatcher. A lot of useful information that is beyond the scope of this article is available, and the search function is quite invaluable when trying to hunt down problems in your cluster.