Did you know the cluvfy healthcheck?
Posted by Martin Bach on January 28, 2012
While I was performing a three day seminar recently in Switzerland I came across this new option in cluvfy.
Normally you’d run cluvfy in preparation of the installation of Grid Infrastructure or a set of RAC binaries to ensure everything is ready for the next step in the RAC install process. Beginning with 11.2.0.3, there is another option that’s been sneaked in without too much advertisement: the healthcheck.
Part of the “comp” checks, it takes the following options:
cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name] [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]
The most extensive report is run without any options, as shown in the appendix (the output is too long to display at this stage of the post) You have the following options:
- collect: you can either specify to collect information about the cluster, or the database. Don’t specify the collect option and the health check will collect information about both. See also the next option if you want to collect database related information!
- db: database unique name to collect information for. Before you can use this option, you need to run the script cvusys.sql in $GRID_HOME/cv/admin. It will create a user the healthcheck tool will connect against to find recommendations. If omitted, all cluster databases registered in the OCR will be probed.
- bestpractice, mandatory, deviations are self explanatory. As always, don’t take the “best practice” literally!
- html: Creates a HTML report in savedir, see below. If not used, a text based output is created
- save, savedir: save the report in savedir
Now this seems to be a nice way to produce a report at the end of the RAC installation which shows that the system is correctly set up.
Appendix
Here is a text version of a full report I generated on my 2 node cluster, comprised of rac11203node1 and rac11203node2. I didn’t have a database ready at the time hence the limitation to the cluster health check.
****************************************************************************************** Summary of environment ****************************************************************************************** Date (mm/dd/yyyy) : 01/28/2012 Time (hh:mm:ss) : 21:57:03 Cluster name : rac11203 Clusterware version : 11.2.0.3.0 Grid home : /u01/crs/11.2.0.3 Grid User : oracle Operating system : Linux2.6.32-100.34.1.el6uek.x86_64 ****************************************************************************************** System requirements ****************************************************************************************** Verification Check : Network Time Protocol (NTP) Verification Description : This task verifies cluster time synchronization on clusters that use Network Time Protocol (NTP). Verification Result : FAILED Cluster-wide error details : Error - If you plan to use CTSS for time synchronization then NTP configuration must be uninstalled on all nodes of the cluster. Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 FAILED not available not available rac11203node1 FAILED not available not available __________________________________________________________________________________________ Verification Check : Swap Size Verification Description : This is a prerequisite condition to test whether sufficient total swap space is available on the system. Verification Result : WARNING Verification Summary : Swap configuration did not meet the recommended value of null on rac11203node2,rac11203node1 Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 FAILED 3.9127GB (4102768.0KB) 1023.9922MB (1048568.0KB) rac11203node1 FAILED 3.9127GB (4102768.0KB) 1023.9922MB (1048568.0KB) __________________________________________________________________________________________ Verification Check : Time zone consistency Verification Description : This task checks for the consistency of time zones across systems. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Task resolv.conf Integrity Verification Description : This task checks consistency of file /etc/resolv.conf file across nodes Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : User Mask Verification Description : This is a prerequisite condition to make sure the user file creation mask (umask) is "0022". Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 0022 0022 rac11203node1 PASSED 0022 0022 __________________________________________________________________________________________ Verification Check : User Not In Group: oracle Verification Description : This is a prerequisite condition to make sure user "oracle" is not part of "root" group. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Same core file name pattern Verification Description : This task checks the consistency of core file name pattern across systems. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Root user consistency Verification Description : This test checks the consistency of the primary group of the root user across the cluster nodes Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Current Group ID Verification Description : This test verifies that the user is currently logged in to the user's primary group. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Users With Same UID Verification Description : This test checks that multiple users do not exist with user id as "0". Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : OS Kernel Version Verification Description : This is a prerequisite condition to test whether the system kernel version is at least "2.6.32". Verification Result : PASSED Verification Summary : Kernel version meets recommendation Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 2.6.32 2.6.32-100.34.1.el6uek.x86_6 4 rac11203node1 PASSED 2.6.32 2.6.32-100.34.1.el6uek.x86_6 4 __________________________________________________________________________________________ Verification Check : Architecture Verification Description : This is a prerequisite condition to test whether the system has a certified architecture. Verification Result : PASSED Verification Summary : Architecture recommendation is met Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED x86_64 x86_64 rac11203node1 PASSED x86_64 x86_64 __________________________________________________________________________________________ Verification Check : Soft Limit: maximum user processes Verification Description : This is a prerequisite condition to test whether the soft limit for "maximum user processes" is set to at least 2047. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 2047 2047 rac11203node1 PASSED 2047 2047 __________________________________________________________________________________________ Verification Check : Hard Limit: maximum user processes Verification Description : This is a prerequisite condition to test whether the hard limit for "maximum user processes" is set to at least 16384. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 16384 16384 rac11203node1 PASSED 16384 16384 __________________________________________________________________________________________ Verification Check : Soft Limit: maximum open file descriptors Verification Description : This is a prerequisite condition to test whether the soft limit for "maximum open file descriptors" is set to at least 1024. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 1024 1024 rac11203node1 PASSED 1024 1024 __________________________________________________________________________________________ Verification Check : Hard Limit: maximum open file descriptors Verification Description : This is a prerequisite condition to test whether the hard limit for "maximum open file descriptors" is set to at least 65536. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 65536 65536 rac11203node1 PASSED 65536 65536 __________________________________________________________________________________________ Verification Check : Run Level Verification Description : This is a prerequisite condition to test whether the system is running with proper run level. Verification Result : PASSED Verification Summary : Run level recommendation are met Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 3,5 3 rac11203node1 PASSED 3,5 3 __________________________________________________________________________________________ Verification Check : Group Membership: oinstall(Primary) Verification Description : This is a prerequisite condition to test whether user "oracle" has group "oinstall" as its primary group. Verification Result : PASSED Verification Summary : Group oinstall is the primary group of user oracle Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Group Existence: oinstall Verification Description : This is a prerequisite condition to test whether group "oinstall" exists on the system. Verification Result : PASSED Verification Summary : Group oinstall exists Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED true true rac11203node1 PASSED true true __________________________________________________________________________________________ Verification Check : User Existence: oracle Verification Description : This is a prerequisite condition to test whether user "oracle" exists on the system. Verification Result : PASSED Verification Summary : User oracle exists Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED true true rac11203node1 PASSED true true __________________________________________________________________________________________ Verification Check : Available Physical Memory Verification Description : This is a prerequisite condition to test whether the system has at least 50MB (51200.0KB) of available physical memory. Verification Result : PASSED Verification Summary : Available memory meets or exceeds recommendation Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 50MB (51200.0KB) 3.029GB (3176136.0KB) rac11203node1 PASSED 50MB (51200.0KB) 2.7401GB (2873240.0KB) __________________________________________________________________________________________ Verification Check : Physical Memory Verification Description : This is a prerequisite condition to test whether the system has at least 1.5GB (1572864.0KB) of total physical memory. Verification Result : PASSED Verification Summary : Physical memory meets or exceeds recommendation Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 1.5GB (1572864.0KB) 3.9127GB (4102768.0KB) rac11203node1 PASSED 1.5GB (1572864.0KB) 3.9127GB (4102768.0KB) __________________________________________________________________________________________ Verification Check : ASMLib installation and configuration verification. Verification Description : This task checks the ASMLib installation and configuration across the systems. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Multicast check Verification Description : This task checks that network interfaces in subnet are able to communicate over multicast IP address Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Node Connectivity Verification Description : This is a prerequisite condition to test whether connectivity exists amongst all the nodes. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ ****************************************************************************************** System recommendations ****************************************************************************************** Verification Check : Reverse path filter setting Verification Description : Checks if reverse path filter setting for all private interconnect network interfaces is correct Verification Result : MET Verification Summary : Check for Reverse path filter setting passed Additional Details : Reverse path filter parameter "rp_filter" must be set to the value of 0 or 2 for all the private interconnect network interfaces, this will disable or relax the filtering and allow Clusterware to function correctly Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 0|2 0|2 rac11203node1 PASSED 0|2 0|2 __________________________________________________________________________________________ Verification Check : availability of port 8888 Verification Description : availability of port 8888 Verification Result : MET Verification Summary : Check for availability of port 8888 passed Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED true true rac11203node1 PASSED true true __________________________________________________________________________________________ Verification Check : Hardware Clock synchronization at shutdown Verification Description : Checks whether Hardware Clock is synchronized with the system clock during system shutdown Verification Result : MET Verification Summary : Check for Hardware Clock synchronization at shutdown passed Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED HWCLOCKSYNC=1 HWCLOCKSYNC=1 rac11203node1 PASSED HWCLOCKSYNC=1 HWCLOCKSYNC=1 __________________________________________________________________________________________ ****************************************************************************************** Clusterware requirements ****************************************************************************************** Verification Check : Clock Synchronization Verification Description : This test checks the Oracle Cluster Time Synchronization Services across the cluster nodes. Verification Result : FAILED Cluster-wide error details : Error - Look at the accompanying error messages for the nodes on which the check failed and fix the problem. If you plan to use CTSS for time synchronization then NTP configuration should be uninstalled on all nodes of the cluster. Refer to "Preparing Your Cluster" of "Oracle Database 2 Day+ Real Application Clusters Guide". Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 FAILED not available not available rac11203node1 FAILED not available not available __________________________________________________________________________________________ Verification Check : Node Connectivity Verification Description : This is a prerequisite condition to test whether connectivity exists amongst all the nodes. The connectivity is being tested for the subnets "192.168.99.0,192.168.100.0,192.168.101.0" Verification Result : FAILED Cluster-wide error details : Error - Verify the interface configurations for the network interfaces identified on the nodes indicated using utilities like ipconfig or ping. Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node1 : FAILED not available not available 192.168.99.27 rac11203node2 FAILED not available not available rac11203node1 FAILED not available not available __________________________________________________________________________________________ Verification Check : VIP Subnet configuration check Verification Description : This task checks that all VIP subnetworks match each other and at least one public network interface of the cluster Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : User Not In Group: oracle Verification Description : This is a prerequisite condition to make sure user "oracle" is not part of "root" group. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : ACFS Integrity Verification Description : This test checks the integrity of Oracle ASM Cluster File System across the cluster nodes. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : OLR Integrity Verification Description : This test checks the integrity of OLR on the local node. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Single Client Access Name (SCAN) Verification Description : This test verifies the Single Client Access Name configuration. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Node Application Existence Verification Description : This test checks the existence of Node Applications on the system. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : CRS Integrity Verification Description : This test checks the integrity of Oracle Clusterware stack across the cluster nodes. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : OCR Integrity Verification Description : This test checks the integrity of OCR across the cluster nodes. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Cluster Integrity Verification Description : This test checks the integrity of the cluster. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : User Mask Verification Description : This is a prerequisite condition to make sure the user file creation mask (umask) is "0022". Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 0022 0022 rac11203node1 PASSED 0022 0022 __________________________________________________________________________________________ Verification Check : UDev attribtes check Verification Description : This is a pre-check condition to check if the devices entries in the Udev permissions file have been set up correctly. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : UDev attribtes check Verification Description : This is a pre-check condition to check if the devices entries in the Udev permissions file have been set up correctly. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Cluster Manager Integrity Verification Description : This test checks the integrity of cluster manager across the cluster nodes. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Voting Disk Verification Description : This test verifies the Oracle Clusterware voting disk configuration which is used to determine which instances are members of a cluster. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Time zone consistency Verification Description : This task checks for the consistency of time zones across systems. Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ Verification Check : Multicast check Verification Description : This task checks that network interfaces in subnet are able to communicate over multicast IP address Verification Result : PASSED Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED not available not available rac11203node1 PASSED not available not available __________________________________________________________________________________________ ****************************************************************************************** Clusterware recommendations ****************************************************************************************** Verification Check : CSS disktimeout parameter Verification Description : Checks if the CSS disktimeout is set correctly on the system Verification Result : MET Verification Summary : Check for CSS disktimeout parameter passed Additional Details : The maximum amount of time allowed for a voting file I/O to complete; if this time is exceeded the voting disk will be marked as offline. Note that this is also the amount of time that will be required for initial cluster formation, i.e. when no nodes have previously been up and in a cluster. References (URLs/Notes) : https://support.oracle.com/CSP/main/article?cmd=show&type=N OT&id=294430.1 Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 200 200 rac11203node1 PASSED 200 200 __________________________________________________________________________________________ Verification Check : CSS reboottime parameter Verification Description : Checks if the CSS reboottime is set correctly on the system Verification Result : MET Verification Summary : Check for CSS reboottime parameter passed Additional Details : reboottime (default 3 seconds) is the amount of time allowed for a node to complete a reboot after the CSS daemon has been evicted. References (URLs/Notes) : https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=294430.1 Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 3 3 rac11203node1 PASSED 3 3 __________________________________________________________________________________________ Verification Check : CSS misscount parameter Verification Description : Checks if the CSS misscount is set correctly on the system Verification Result : MET Verification Summary : Check for CSS misscount parameter passed Additional Details : The CSS misscount parameter represents the maximum time, in seconds, that a network heartbeat can be missed before entering into a cluster reconfiguration to evict the node References (URLs/Notes) : https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=294430.1 Node Status Expected Value Actual Value ------------------------------------------------------------------------------------------ rac11203node2 PASSED 30 30 rac11203node1 PASSED 30 30 __________________________________________________________________________________________
As this is lab kit, I don’t worry about the NTP and SWAP errors-you however should!
Wissem said
Hi Martin,
Is it similar to RACCheck tool output?
Thanks,
Wissem
Martin Bach said
I have never run RACCheck-let me test and get back to you.
Martin Bach said
OK, I have now run it. RACCheck is different (it knows 10.1 up to 11.2), and also produces different output. RACCheck requires root privileges as well for certain checks, so I suppose it’s more detailed than the healthcheck. IMO the biggest advantage of the healthcheck tool is the ability to be run without any installation needed (RACCheck needs to be deployed) and not requiring root privileges.
Anand said
Hi Matrin,
Thanks for sharing it. Wanted to know what command did u use to generate this output. I am trying ./cluvfy comp healthcheck -collect cluster -bestpractice, it doesn;t list so much of details in the output.
Regards,
Anand
Martin Bach said
Hi Anand,
I think it was cluvfy comp healthcheck -collect cluster -save
Regards,
Martin