Result! I have finally been able to gather a complete RDA (Oracle Remote Diagnostic Agent) output on my 2 node RAC system. After consulting the relevant documentation on MOS-which is spread over at least 42 Doc IDs-I found them not to be very helpful to the degree that some of what I read is actually wrong or contradicting. I put together a short note, primarily to myself, to remind me of the process. I hope you find it useful, too.
The RDA version I used for this post is 8.14.x from MOS March 4th 2017. My RAC nodes are based on Oracle Linux 7.3/UEK 4.
Starting the data collection
RDA is one of the standard tools I use and I have previously blogged about it. RDA started off as a simple-to-use tool. After having used it for some years I started to run into issues with the interactive configuration which simply took too long to complete. As soon as I learned about them I fell in love with profiles. RDA profiles only prompt you for relevant information about the product you want to collect data for.
Running RDA on a RAC system is similar to single instance, and you automatically use profiles-a nice touch. It appears to me that RAC data collection is triggered via SSH on the remote nodes with the results being transferred to the local node.
I believe there are two methods of data gathering on RAC: one for Grid Infrastructure, and another one for the RDBMS layer. Depending on your settings of ORACLE_HOME and ORACLE_SID different parts of the stack are analysed. In this post I was primarily interested in running RDA for the RDBMS part of my cluster.
I started off by downloading the current RDA version to the system I want to run it on. This is a 2-node RAC, with nodes named rac12pri1 and rac12pri2. Both are based on Oracle Linux 7.3 with the most current UEK4 at the time of writing. My RDBMS homes are 12.1.0.2 patched with the January 2017 proactive bundle patch + OJVM.
The first caveat looms right at the beginning of the entire process: I didn’t find it stated explicitly in the documentation on MOS, but it seemed that you need to deploy RDA on all nodes in the cluster, and in the exact same location, before you start the data collection. I use /home/oracle/rda for that purpose. Make sure you have plenty of space in the location you chose, sometimes /home doesn’t provide enough for larger, more active systems.
During my first unsuccessful attempts I didn’t deploy RDA on all nodes before starting the data collection, only to see it not gather any information from the remote node. This is somewhat confusing, because the output (when collecting data) states this:
NOD002: Installing RDA software
NOD002 is short for the second node in my cluster.
Also, when configuring the data gathering process, you will see a prompt like this one:
Enter the absolute path of the directory where RDA will be installed on the remote nodes.
To the casual observer like me these messages suggest that RDA is actually installed on the remote nodes as part of the data collection process-but it is only partially done. Comparing directory sizes between remote and local node revealed /home/oracle/rda to be greater than 200M locally, while on the remote nodes it was only 37MB in size. Something seems to be missing…
Once deployed, you can change to the RDA directory and prepare for the data collection. You actually execute RDA on the first node, the rest of the work is done programatically. The MOS note seems to be correct this time, here is an example of my configuration session:
[oracle@rac12pri1 rda]$ ./rda.pl -vXRda start CLOUD -pRac_Assessment
Creating collection "output" ...
- Trying to identify the domain ... (can take time)
Inside DFT scenario, define the input and profile targets
Inside DFT scenario, check Oracle home or Middleware presence
- RDA:DCbegin ...
------------------------------------------------------------------------------
RDA.BEGIN: Initializes the Data Collection
------------------------------------------------------------------------------
Enter the Oracle home to be used for data analysis
Press Return to accept the default (/u01/app/oracle/product/12.1.0.2/dbhome_1)
>
- RDA:DCconfig ...
------------------------------------------------------------------------------
RDA.CONFIG: Collects Key Configuration Information
------------------------------------------------------------------------------
- RDA:DCocm ...
------------------------------------------------------------------------------
RDA.OCM: Set up the Configuration Manager Interface
------------------------------------------------------------------------------
- RDA:DCstatus ...
------------------------------------------------------------------------------
RDA.STATUS: Produces the Remote Data Collection Reports
------------------------------------------------------------------------------
- RDA:DCload ...
------------------------------------------------------------------------------
RDA.LOAD: Produces the External Collection Reports
------------------------------------------------------------------------------
- RDA:DCfilter ...
------------------------------------------------------------------------------
RDA.FILTER: Controls Report Content Filtering
------------------------------------------------------------------------------
- RDA:DCend ...
------------------------------------------------------------------------------
RDA.END: Finalizes the Data Collection
------------------------------------------------------------------------------
In CRS start scenario, getting nodes using /u01/app/12.1.0.2/grid/bin/olsnodes
2>/dev/null
------------------------------------------------------------------------------
Requesting common cluster information
------------------------------------------------------------------------------
Enter the absolute path of the directory where RDA will be installed on the
remote nodes.
Press Return to accept the default (/home/oracle/rda)
>
Do you want RDA to use an alternative login to execute remote requests (Y/N)?
Press Return to accept the default (N)
>
In CRS start scenario, getting local node information
In CRS start scenario, pinging node rac12pri2
------------------------------------------------------------------------------
Requesting information for node rac12pri1
------------------------------------------------------------------------------
Enter the Oracle home to be analyzed on the node rac12pri1
Press Return to accept the default (/u01/app/oracle/product/12.1.0.2/dbhome_1)
>
Enter the Oracle SID to be analyzed on the node rac12pri1
Press Return to accept the default (NCDBA1)
>
------------------------------------------------------------------------------
Requesting information for node rac12pri2
------------------------------------------------------------------------------
Enter the Oracle home to be analyzed on the node rac12pri2
Press Return to accept the default (/u01/app/oracle/product/12.1.0.2/dbhome_1)
>
Enter the Oracle SID to be analyzed on the node rac12pri2
> NCDBA2
- RDA:DCremote ...
------------------------------------------------------------------------------
RAC Setup Summary
------------------------------------------------------------------------------
Nodes:
. NOD001 rac12pri1/NCDBA1
. NOD002 rac12pri2/NCDBA2
2 nodes found
So RDA understands the RAC scenario, and it gathers data for each node that is part of the cluster as reported by the olsnodes tool. There was nothing really surprising about the prompts, except that I haven’t found a way to analyse more than one database, or ASM and a database together.
Be careful to enter the correct ORACLE_SID for your (remote) RAC nodes. RDA picks up your ORACLE_HOME and ORACLE_SID if they are set.
Optionally verify the correct number of nodes was detected
I am only mentioning this for the sake of completeness, the RAC Setup Summary you saw in the previous step was more than adequate for me. If you really want to find out about the number of nodes you are about to collect information, be careful: MOS Doc ID 359395.1 is wrong – the command ./rda.pl -vC RDA.REMTOE list will not only list the nodes, it will also start the data collection. Use this one instead, which I found in MOS Doc ID 1682909.1:
[oracle@rac12pri1 rda]$ ./rda.pl -XRemote list
Defined nodes:
NOD001 rac12pri1 NCDBA1
NOD002 rac12pri2 NCDBA2
[oracle@rac12pri1 rda]$
It merely lists the nodes, without actually starting to do any work.
Initiate data collection
When you are happy with the output and configuration, start collecting data. Here is an example of my session:
[oracle@rac12pri1 rda]$ ./rda.pl -v -e TRC/TRACE=1
Collecting diagnostic data ...
------------------------------------------------------------------------------
RDA Data Collection Started 04-Mar-2017 16:10:15
------------------------------------------------------------------------------
Processing RDA.BEGIN module ...
Inside BEGIN module, testing the RDA engine code build
Inside BEGIN module, testing the report directory
Inside BEGIN module, testing the module targets
Inside BEGIN module, launching parallel executions
Processing RDA.CONFIG module ...
Inside CONFIG module, listing Oracle homes
Inside CONFIG module, getting Oracle home inventory (can take time)
Processing RDA.REMOTE module ...
NOD001: Detecting storage type
NOD002: Detecting storage type
NOD001: Running RDA command
NOD002: Installing RDA software
NOD002: Running RDA command
NOD002: Transfering report package
NOD001: Transfering report package
Processing RDA.END module ...
Inside END module, gathering system information
Inside END module, getting CPU information (linux)
Inside END module, getting memory information (linux)
Inside END module, producing the file catalog
Inside END module, producing target overview
Inside END module, waiting for parallel execution completion
Inside END module, producing setting overview
------------------------------------------------------------------------------
RDA Data Collection Ended 04-Mar-2017 16:17:44
------------------------------------------------------------------------------
Generating the reports ...
- collect/RDA_CONFIG_homes.txt ...
- collect/RDA_CONFIG_oh_inv.txt ...
- collect/RDA_END_system.txt ...
- collect/RDA_END_files.txt ...
- collect/RDA_END_target.txt ...
- collect/RDA_END_report.txt ...
- Report index ...
Packaging collection results ...
[...skipping a lot of text...]
You can review the reports by transferring the /home/oracle/rda/output
directory structure to a location where you have web-browser access.
Then, point your browser at this file to display the reports:
RDA__start.htm
[...skipping a lot of text...]
It is crucially important to see these lines:
Processing RDA.REMOTE module ...
NOD001: Detecting storage type
NOD002: Detecting storage type
NOD001: Running RDA command
NOD002: Installing RDA software
NOD002: Running RDA command
NOD002: Transfering report package
NOD001: Transfering report package
In my first attempts, when I didn’t deploy RDA on all nodes myself, the lines “NOD002: Running RDA command” and “NOD002: Transfering report package” weren’t shown. Unsurprisingly no data was gathered on the remote nodes.
Viewing the output
At the end of the data collection you should see a *.tar.gz file per node. In my 2 node cluster setup, there are two:
[oracle@rac12pri1 rda]$ ls output/remote/
RDA_nod001_output.tar.gz RDA_nod002_output.tar.gz
You can view these after extracting to a temporary location in your browser, start with the file named RDA__start.htm which presents the various parts of the report.
Further reading: Testing user equivalence prior to gathering data
While trying to troubleshoot my remote data gathering problems (I suspected a problem with SSH at first) I noticed that RDA offers test modules as well (see MOS Doc ID: 330760.1). What’s missing from the Doc ID is an example on how to invoke the SSH test module, or rather the command in the RAC specific instructions seems not to work. But it isn’t too hard to figure out the proper call to the RDA executable. The following worked for me:
[oracle@rac12pri1 rda]$ ./rda.pl -T ssh
Processing remote operation test module ...
Command Availability:
rcp (Not found)
remsh (Not found)
rsh (Not found)
scp /bin/scp
ssh /bin/ssh
ssh-add /bin/ssh-add
ssh-agent /bin/ssh-agent
Related files:
/etc/ssh/moduli
/etc/ssh/ssh_config
/etc/ssh/ssh_host_ecdsa_key
/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub
/etc/ssh/sshd_config
/home/oracle/.ssh/authorized_keys
/home/oracle/.ssh/config
/home/oracle/.ssh/config.backup
/home/oracle/.ssh/id_rsa
/home/oracle/.ssh/id_rsa.pub
/home/oracle/.ssh/known_hosts
Check if an authentication agent has been started
SSH_AGENT_PID=1
SSH_AUTH_SOCK=1
Agent identities:
1024 some:colums:separated:by:colons:indicating:the:ID
/home/oracle/.ssh/id_rsa (RSA)
Driver Availability:
da Available
jsch Available
ssh Available
rsh -
Settings modified by the remote operation initialization:
REMOTE.F_SCP_COMMAND=/bin/scp
REMOTE.F_SSH_COMMAND=/bin/ssh
REMOTE.T_SCP_OPTIONS=-BCpq -o ConnectTimeout=30
REMOTE.T_SSH_OPTIONS=-Cnq -o ConnectTimeout=30
------------------------------------------------------------------------------
Test a remote command
------------------------------------------------------------------------------
Enter the remote host name
Press Return to accept the default (localhost)
> rac12pri2
Do you want RDA to use an alternative login to execute remote requests (Y/N)?
Press Return to accept the default (N)
>
Check remote command execution with best driver
Exit code: 0
Check remote command execution using DA
Exit code: 0
Check remote command execution using JSCH
Exit code: 0
Check remote command execution using SSH
Exit code: 0
Check remote command execution using SSH (no timeout)
Exit code: 0
[oracle@rac12pri1 rda]$
Looking at the above output let me to believe that there isn’t a SSH-related problem with my cluster.
Martins Blog 
You must be logged in to post a comment.