Martins Blog

Trying to explain complex things in simple terms

Did you know the cluvfy healthcheck?

Posted by Martin Bach on January 28, 2012

While I was performing a three day seminar recently in Switzerland I came across this new option in cluvfy.

Normally you’d run cluvfy in preparation of the installation of Grid Infrastructure or a set of RAC binaries to ensure everything is ready for the next step in the RAC install process. Beginning with 11.2.0.3, there is another option that’s been sneaked in without too much advertisement: the healthcheck.

Part of the “comp” checks, it takes the following options:

cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name] [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]

The most extensive report is run without any options, as shown in the appendix (the output is too long to display at this stage of the post) You have the following options:

  • collect: you can either specify to collect information about the cluster, or the database. Don’t specify the collect option and the health check will collect information about both. See also the next option if you want to collect database related information!
  • db: database unique name to collect information for. Before you can use this option, you need to run the script cvusys.sql in $GRID_HOME/cv/admin. It will create a user the healthcheck tool will connect against to find recommendations. If omitted, all cluster databases registered in the OCR will be probed.
  • bestpractice, mandatory, deviations are self explanatory. As always, don’t take the “best practice” literally!
  • html: Creates a HTML report in savedir, see below. If not used, a text based output is created
  • save, savedir: save the report in savedir

Now this seems to be a nice way to produce a report at the end of the RAC installation which shows that the system is correctly set up.

Appendix

Here is a text version of a full report I generated on my 2 node cluster, comprised of rac11203node1 and rac11203node2. I didn’t have a database ready at the time hence the limitation to the cluster health check.


******************************************************************************************
Summary of environment
******************************************************************************************

Date (mm/dd/yyyy)    :  01/28/2012
Time (hh:mm:ss)      :  21:57:03
Cluster name         :  rac11203
Clusterware version  :  11.2.0.3.0
Grid home            :  /u01/crs/11.2.0.3
Grid User            :  oracle
Operating system     :  Linux2.6.32-100.34.1.el6uek.x86_64

******************************************************************************************
System requirements
******************************************************************************************

Verification Check          :  Network Time Protocol (NTP)
Verification Description    :  This task verifies cluster time synchronization on
clusters that use Network Time Protocol (NTP).
Verification Result         :  FAILED
Cluster-wide error details  :  Error -  If you plan to use CTSS for time
synchronization then NTP configuration must be
uninstalled on all nodes of the cluster.

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       FAILED    not available                 not available
rac11203node1       FAILED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Swap Size
Verification Description  :  This is a prerequisite condition to test whether
sufficient total swap space is available on the system.
Verification Result       :  WARNING
Verification Summary      :  Swap configuration did not meet the recommended value of
null on rac11203node2,rac11203node1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       FAILED    3.9127GB (4102768.0KB)        1023.9922MB (1048568.0KB)
rac11203node1       FAILED    3.9127GB (4102768.0KB)        1023.9922MB (1048568.0KB)

__________________________________________________________________________________________

Verification Check        :  Time zone consistency
Verification Description  :  This task checks for the consistency of time zones across
systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Task resolv.conf Integrity
Verification Description  :  This task checks consistency of file /etc/resolv.conf file
across nodes
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  User Mask
Verification Description  :  This is a prerequisite condition to make sure the user
file creation mask (umask) is "0022".
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    0022                          0022
rac11203node1       PASSED    0022                          0022

__________________________________________________________________________________________

Verification Check        :  User Not In Group: oracle
Verification Description  :  This is a prerequisite condition to make sure user
"oracle" is not part of "root" group.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Same core file name pattern
Verification Description  :  This task checks the consistency of core file name pattern
across systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Root user consistency
Verification Description  :  This test checks the consistency of the primary group of
the root user across the cluster nodes
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Current Group ID
Verification Description  :  This test verifies that the user is currently logged in to
the user's primary group.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Users With Same UID
Verification Description  :  This test checks that multiple users do not exist with
user id as "0".
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  OS Kernel Version
Verification Description  :  This is a prerequisite condition to test whether the
system kernel version is at least "2.6.32".
Verification Result       :  PASSED
Verification Summary      :  Kernel version meets recommendation

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    2.6.32                        2.6.32-100.34.1.el6uek.x86_6
4
rac11203node1       PASSED    2.6.32                        2.6.32-100.34.1.el6uek.x86_6
4

__________________________________________________________________________________________

Verification Check        :  Architecture
Verification Description  :  This is a prerequisite condition to test whether the
system has a certified architecture.
Verification Result       :  PASSED
Verification Summary      :  Architecture recommendation is met

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    x86_64                        x86_64
rac11203node1       PASSED    x86_64                        x86_64

__________________________________________________________________________________________

Verification Check        :  Soft Limit: maximum user processes
Verification Description  :  This is a prerequisite condition to test whether the soft
limit for "maximum user processes" is set to at least 2047.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    2047                          2047
rac11203node1       PASSED    2047                          2047

__________________________________________________________________________________________

Verification Check        :  Hard Limit: maximum user processes
Verification Description  :  This is a prerequisite condition to test whether the hard
limit for "maximum user processes" is set to at least
16384.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    16384                         16384
rac11203node1       PASSED    16384                         16384

__________________________________________________________________________________________

Verification Check        :  Soft Limit: maximum open file descriptors
Verification Description  :  This is a prerequisite condition to test whether the soft
limit for "maximum open file descriptors" is set to at
least 1024.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    1024                          1024
rac11203node1       PASSED    1024                          1024

__________________________________________________________________________________________

Verification Check        :  Hard Limit: maximum open file descriptors
Verification Description  :  This is a prerequisite condition to test whether the hard
limit for "maximum open file descriptors" is set to at
least 65536.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    65536                         65536
rac11203node1       PASSED    65536                         65536

__________________________________________________________________________________________

Verification Check        :  Run Level
Verification Description  :  This is a prerequisite condition to test whether the
system is running with proper run level.
Verification Result       :  PASSED
Verification Summary      :  Run level recommendation are met

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    3,5                           3
rac11203node1       PASSED    3,5                           3

__________________________________________________________________________________________

Verification Check        :  Group Membership: oinstall(Primary)
Verification Description  :  This is a prerequisite condition to test whether user
"oracle" has group "oinstall" as its primary group.
Verification Result       :  PASSED
Verification Summary      :  Group oinstall is the primary group of user oracle

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Group Existence: oinstall
Verification Description  :  This is a prerequisite condition to test whether group
"oinstall" exists on the system.
Verification Result       :  PASSED
Verification Summary      :  Group oinstall exists

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    true                          true
rac11203node1       PASSED    true                          true

__________________________________________________________________________________________

Verification Check        :  User Existence: oracle
Verification Description  :  This is a prerequisite condition to test whether user
"oracle" exists on the system.
Verification Result       :  PASSED
Verification Summary      :  User oracle exists

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    true                          true
rac11203node1       PASSED    true                          true

__________________________________________________________________________________________

Verification Check        :  Available Physical Memory
Verification Description  :  This is a prerequisite condition to test whether the
system has at least 50MB (51200.0KB) of available physical
memory.
Verification Result       :  PASSED
Verification Summary      :  Available memory meets or exceeds recommendation

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    50MB (51200.0KB)              3.029GB (3176136.0KB)
rac11203node1       PASSED    50MB (51200.0KB)              2.7401GB (2873240.0KB)

__________________________________________________________________________________________

Verification Check        :  Physical Memory
Verification Description  :  This is a prerequisite condition to test whether the
system has at least 1.5GB (1572864.0KB) of total physical
memory.
Verification Result       :  PASSED
Verification Summary      :  Physical memory meets or exceeds recommendation

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    1.5GB (1572864.0KB)           3.9127GB (4102768.0KB)
rac11203node1       PASSED    1.5GB (1572864.0KB)           3.9127GB (4102768.0KB)

__________________________________________________________________________________________

Verification Check        :  ASMLib installation and configuration verification.
Verification Description  :  This task checks the ASMLib installation and configuration
across the systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Multicast check
Verification Description  :  This task checks that network interfaces in subnet are
able to communicate over multicast IP address
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Node Connectivity
Verification Description  :  This is a prerequisite condition to test whether
connectivity exists amongst all the nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________


******************************************************************************************
System recommendations
******************************************************************************************

Verification Check        :  Reverse path filter setting
Verification Description  :  Checks if reverse path filter setting for all private
interconnect network interfaces is correct
Verification Result       :  MET
Verification Summary      :  Check for Reverse path filter setting passed
Additional Details        :  Reverse path filter parameter "rp_filter" must be set to
the value of 0 or 2 for all the private interconnect
network interfaces, this will disable or relax the
filtering and allow Clusterware to function correctly

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    0|2                           0|2
rac11203node1       PASSED    0|2                           0|2

__________________________________________________________________________________________

Verification Check        :  availability of port 8888
Verification Description  :  availability of port 8888
Verification Result       :  MET
Verification Summary      :  Check for availability of port 8888 passed

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    true                          true
rac11203node1       PASSED    true                          true

__________________________________________________________________________________________

Verification Check        :  Hardware Clock synchronization at shutdown
Verification Description  :  Checks whether Hardware Clock is synchronized with the
system clock during system shutdown
Verification Result       :  MET
Verification Summary      :  Check for Hardware Clock synchronization at shutdown passed

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    HWCLOCKSYNC=1                 HWCLOCKSYNC=1
rac11203node1       PASSED    HWCLOCKSYNC=1                 HWCLOCKSYNC=1

__________________________________________________________________________________________


******************************************************************************************
Clusterware requirements
******************************************************************************************

Verification Check          :  Clock Synchronization
Verification Description    :  This test checks the Oracle Cluster Time Synchronization
Services across the cluster nodes.
Verification Result         :  FAILED
Cluster-wide error details  :  Error -  Look at the accompanying error messages for the
nodes on which the check failed and fix the problem. If
you plan to use CTSS for time synchronization then NTP
configuration should be uninstalled on all nodes of the
cluster. Refer to "Preparing Your Cluster" of "Oracle
Database 2 Day+ Real Application Clusters Guide".

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       FAILED    not available                 not available
rac11203node1       FAILED    not available                 not available

__________________________________________________________________________________________

Verification Check          :  Node Connectivity
Verification Description    :  This is a prerequisite condition to test whether
connectivity exists amongst all the nodes. The
connectivity is being tested for the subnets
"192.168.99.0,192.168.100.0,192.168.101.0"
Verification Result         :  FAILED
Cluster-wide error details  :  Error -  Verify the interface configurations for the
network interfaces identified on the nodes indicated
using utilities like ipconfig or ping.

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node1 :     FAILED    not available                 not available
192.168.99.27
rac11203node2       FAILED    not available                 not available
rac11203node1       FAILED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  VIP Subnet configuration check
Verification Description  :  This task checks that all VIP subnetworks match each other
and at least one public network interface of the cluster
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  User Not In Group: oracle
Verification Description  :  This is a prerequisite condition to make sure user
"oracle" is not part of "root" group.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  ACFS Integrity
Verification Description  :  This test checks the integrity of Oracle ASM Cluster File
System across the cluster nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  OLR Integrity
Verification Description  :  This test checks the integrity of OLR on the local node.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Single Client Access Name (SCAN)
Verification Description  :  This test verifies the Single Client Access Name
configuration.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Node Application Existence
Verification Description  :  This test checks the existence of Node Applications on the
system.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  CRS Integrity
Verification Description  :  This test checks the integrity of Oracle Clusterware stack
across the cluster nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  OCR Integrity
Verification Description  :  This test checks the integrity of OCR across the cluster
nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Cluster Integrity
Verification Description  :  This test checks the integrity of the cluster.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  User Mask
Verification Description  :  This is a prerequisite condition to make sure the user
file creation mask (umask) is "0022".
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    0022                          0022
rac11203node1       PASSED    0022                          0022

__________________________________________________________________________________________

Verification Check        :  UDev attribtes check
Verification Description  :  This is a pre-check condition to check if the devices
entries in the Udev permissions file have been set up
correctly.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  UDev attribtes check
Verification Description  :  This is a pre-check condition to check if the devices
entries in the Udev permissions file have been set up
correctly.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Cluster Manager Integrity
Verification Description  :  This test checks the integrity of cluster manager across
the cluster nodes.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Voting Disk
Verification Description  :  This test verifies the Oracle Clusterware voting disk
configuration which is used to determine which instances
are members of a cluster.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Time zone consistency
Verification Description  :  This task checks for the consistency of time zones across
systems.
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________

Verification Check        :  Multicast check
Verification Description  :  This task checks that network interfaces in subnet are
able to communicate over multicast IP address
Verification Result       :  PASSED

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    not available                 not available
rac11203node1       PASSED    not available                 not available

__________________________________________________________________________________________


******************************************************************************************
Clusterware recommendations
******************************************************************************************

Verification Check        :  CSS disktimeout parameter
Verification Description  :  Checks if the CSS disktimeout is set correctly on the
system
Verification Result       :  MET
Verification Summary      :  Check for CSS disktimeout parameter passed
Additional Details        :  The maximum amount of time allowed for a voting file I/O
to complete; if this time is exceeded the voting disk will
be marked as offline.  Note that this is also the amount
of time that will be required for initial cluster
formation, i.e. when no nodes have previously been up and
in a cluster.
References (URLs/Notes)   :  https://support.oracle.com/CSP/main/article?cmd=show&type=N
OT&id=294430.1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    200                           200
rac11203node1       PASSED    200                           200

__________________________________________________________________________________________

Verification Check        :  CSS reboottime parameter
Verification Description  :  Checks if the CSS reboottime is set correctly on the system
Verification Result       :  MET
Verification Summary      :  Check for CSS reboottime parameter passed
Additional Details        :  reboottime (default 3 seconds) is the amount of time
allowed for a node to complete a reboot after the CSS
daemon has been evicted.
References (URLs/Notes)   :  https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=294430.1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    3                             3
rac11203node1       PASSED    3                             3

__________________________________________________________________________________________

Verification Check        :  CSS misscount parameter
Verification Description  :  Checks if the CSS misscount is set correctly on the system
Verification Result       :  MET
Verification Summary      :  Check for CSS misscount parameter passed
Additional Details        :  The CSS misscount parameter represents the maximum time,
in seconds, that a network heartbeat can be missed before
entering into a cluster reconfiguration to evict the node
References (URLs/Notes)   :  https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=294430.1

Node                Status    Expected Value                Actual Value
------------------------------------------------------------------------------------------

rac11203node2       PASSED    30                            30
rac11203node1       PASSED    30                            30

__________________________________________________________________________________________

As this is lab kit, I don’t worry about the NTP and SWAP errors-you however should!

5 Responses to “Did you know the cluvfy healthcheck?”

  1. Wissem said

    Hi Martin,

    Is it similar to RACCheck tool output?
    Thanks,
    Wissem

    • Martin Bach said

      I have never run RACCheck-let me test and get back to you.

      • Martin Bach said

        OK, I have now run it. RACCheck is different (it knows 10.1 up to 11.2), and also produces different output. RACCheck requires root privileges as well for certain checks, so I suppose it’s more detailed than the healthcheck. IMO the biggest advantage of the healthcheck tool is the ability to be run without any installation needed (RACCheck needs to be deployed) and not requiring root privileges.

  2. Anand said

    Hi Matrin,

    Thanks for sharing it. Wanted to know what command did u use to generate this output. I am trying ./cluvfy comp healthcheck -collect cluster -bestpractice, it doesn;t list so much of details in the output.

    Regards,
    Anand

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: