Martins Blog

Trying to explain complex things in simple terms

Moving OCR and voting disks to block devices in CRS 10.2

Posted by Martin Bach on May 26, 2010

Anyone to suggest a shorter title for this: you are welcome to comment! Initially I wanted to have this in the first of today’s posts, but got side tracked. So here’s the next step in the CRS 10.2.0.5 installation: moving off the deprecated raw devices to block devices. This is something you can do with OUI in 11.1, but in 10.2 it either requires raw devices or a clustered file system. Here’s the step by step protocol of my session.

The Setup

Before talking about what I am doing a little background information is in order. This example uses single pathing, the difference to multipathing is marginal-just remember to use the logical device names instead of the physical ones. In device mapper multipathing this might be similar to /dev/mapper/ocrvotep{5,6,7,8,9} in case you map your WWID for the OCR and voting disks to a logical name called “ocrvote”. If presenting only 1 device to the server to store all copies of OCR and voting disk make sure to have it well protected on the array! With this said, here is my setup. First for the voting disk-it went to /dev/raw/raw2 during the installation:

[root@crs102node1 bin]# ./crsctl query css votedisk
 0.     0    /dev/raw/raw2

located 1 votedisk(s).

My OCR doesn’t have a mirror and it on /dev/raw/raw1

[root@crs102node1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
 Version                  :          2
 Total space (kbytes)     :    1003728
 Used space (kbytes)      :       2008
 Available space (kbytes) :    1001720
 ID                       : 1189351742
 Device/File Name         : /dev/raw/raw1
 Device/File integrity check succeeded

 Device/File not configured

 Cluster registry integrity check succeeded

Now let’s change directory to $ORA_CRS_HOME/bin before the action starts.

Voting Disks to Block Device

[root@crs102node1 ~]# cd /u01/crs/oracle/product/10.2.0/crs/bin

To maintain voting disks it has always been a good idea to stop the cluster on ALL nodes.

[root@crs102node1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@crs102node1 bin]# ./crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM 

Ok, that’s definitely down. I repeated this on the second node:

[root@crs102node2 ~]# cd /u01/crs/oracle/product/10.2.0/crs/bin
[root@crs102node2 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@crs102node2 bin]# ./crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM 

I had extra raw devices mapped (raw{3,4,5}) , need to get rid of them first. Note that no restart is needed for this!

[root@crs102node1 bin]# raw -qa
/dev/raw/raw1:    bound to major 202, minor 37
/dev/raw/raw2:    bound to major 202, minor 38
/dev/raw/raw3:    bound to major 202, minor 39
/dev/raw/raw4:    bound to major 202, minor 40
/dev/raw/raw5:    bound to major 202, minor 41

I didn’t use udev for the raw device mapping since I knew they wouldn’t stay past a reboot. I opted for the quick and dirty /etc/rc.local method as in this example:

cat /etc/rc.local
# raw devices
raw /dev/raw/raw1 /dev/xvdc5
raw /dev/raw/raw2 /dev/xvdc6
raw /dev/raw/raw3 /dev/xvdc7
raw /dev/raw/raw4 /dev/xvdc8
raw /dev/raw/raw5 /dev/xvdc9

chown oracle:dba /dev/raw/raw*
chmod 664 /dev/raw/raw*

By assigning raw devices to major 0/minor 0 we unbind them as the following example demonstrates:

[root@crs102node1 bin]# raw /dev/raw/raw3 0 0
/dev/raw/raw3:    bound to major 0, minor 0
[root@crs102node1 bin]# raw /dev/raw/raw4 0 0
/dev/raw/raw4:    bound to major 0, minor 0
[root@crs102node1 bin]# raw /dev/raw/raw5 0 0
/dev/raw/raw5:    bound to major 0, minor 0

Only the required two are left. I did the same on the second node.

[root@crs102node1 bin]# raw -qa
/dev/raw/raw1:    bound to major 202, minor 37
/dev/raw/raw2:    bound to major 202, minor 38

This is my disk layout:

[root@crs102node1 bin]# fdisk -l /dev/xvdc

Disk /dev/xvdc: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

 Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1               1         652     5237158+   5  Extended
/dev/xvdc5               1         125     1003999+  83  Linux
/dev/xvdc6             126         250     1004031   83  Linux
/dev/xvdc7             251         375     1004031   83  Linux
/dev/xvdc8             376         500     1004031   83  Linux
/dev/xvdc9             501         625     1004031   83  Linux

The current mapping is:

/dev/raw/raw1 -> /dev/xvdc5
/dev/raw/raw2 -> /dev/xvdc6

This means that I can make use of /dev/xvdc{7,8,9} for some juggling around. Let’s get started. The plan is to add vote disk for /dev/xvdc{6,7} to have 3 voting disks in total. Remember that we need an odd number of voting disks!

[root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc6
Cluster is not in a ready state for online disk addition

Ooops, but luckily there is a force option:

[root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc6 -force
Now formatting voting disk: /dev/xvdc6
successful addition of votedisk /dev/xvdc6.
[root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc7 -force
Now formatting voting disk: /dev/xvdc7
successful addition of votedisk /dev/xvdc7.

[root@crs102node1 bin]# ./crsctl query css votedisk
 0.     0    /dev/raw/raw2
 1.     0    /dev/xvdc6
 2.     0    /dev/xvdc7

located 3 votedisk(s).

This makes it 3 votedisks, but still one on a raw device. Let’s solve this-first unbind the raw device, then add yet another vote disk and remove the raw device from the configuration:

[root@crs102node1 bin]# raw /dev/raw/raw2 0 0
/dev/raw/raw2:    bound to major 0, minor 0

[root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc8 -force
Now formatting voting disk: /dev/xvdc8
successful addition of votedisk /dev/xvdc8.

[root@crs102node1 bin]# ./crsctl delete css votedisk /dev/raw/raw2 -force
successful deletion of votedisk /dev/raw/raw2.

We are almost there, let’s check:

[root@crs102node1 bin]# ./crsctl query css votedisk
 0.     0    /dev/xvdc6
 1.     0    /dev/xvdc7
 2.     0    /dev/xvdc8

located 3 votedisk(s).

Brilliant. Get ready to start the cluster-the OCR maintenance can be done while it’s online. First we need to set the permissions correctly; I added this to /etc/rc.local. The more elegant way is to use udev for this.

[root@crs102node1 bin]# chown oracle:dba /dev/xvdc{5,6,7,8,9}
[root@crs102node1 bin]# chmod 664 /dev/xvdc{5,6,7,8,9}
[root@crs102node1 bin]# ls -l /dev/xvdc{5,6,7,8,9}
brw-rw-r-- 1 oracle dba 202, 37 May 25 16:25 /dev/xvdc5
brw-rw-r-- 1 oracle dba 202, 38 May 26 15:39 /dev/xvdc6
brw-rw-r-- 1 oracle dba 202, 39 May 26 15:39 /dev/xvdc7
brw-rw-r-- 1 oracle dba 202, 40 May 26 15:41 /dev/xvdc8
brw-rw-r-- 1 oracle dba 202, 41 May 25 16:25 /dev/xvdc9

Repeat on the other node:

[root@crs102node2 bin]# chown oracle:dba /dev/xvdc{5,6,7,8,9}
[root@crs102node2 bin]# chmod 664 /dev/xvdc{5,6,7,8,9}
[root@crs102node2 bin]# ls -l /dev/xvdc{5,6,7,8,9}
brw-rw-r-- 1 oracle dba 202, 37 May 25 16:26 /dev/xvdc5
brw-rw-r-- 1 oracle dba 202, 38 May 25 16:26 /dev/xvdc6
brw-rw-r-- 1 oracle dba 202, 39 May 25 16:26 /dev/xvdc7
brw-rw-r-- 1 oracle dba 202, 40 May 25 16:26 /dev/xvdc8
brw-rw-r-- 1 oracle dba 202, 41 May 25 16:26 /dev/xvdc9

[root@crs102node2 bin]# raw /dev/raw/raw2 0 0
/dev/raw/raw2:    bound to major 0, minor 0

Now start the cluster on both nodes

[root@crs102node1 bin]# ./crsctl start crs
Attempting to start CRS stack 
The CRS stack will be started shortly

[root@crs102node2 bin]# ./crsctl start crs
Attempting to start CRS stack 
The CRS stack will be started shortly
Additional information about success is in $ORA_CRS_HOME/log/hostname/alert_hostname.log:

2010-05-26 15:29:58.556
[cssd(21486)]CRS-1603:CSSD on node crs102node2 shutdown by user.
2010-05-26 15:44:38.144
[cssd(10993)]CRS-1605:CSSD voting file is online: /dev/xvdc6. ...
2010-05-26 15:44:38.147
[cssd(10993)]CRS-1605:CSSD voting file is online: /dev/xvdc7. ...
2010-05-26 15:44:38.149
[cssd(10993)]CRS-1605:CSSD voting file is online: /dev/xvdc8. ...
2010-05-26 15:44:43.993
[cssd(10993)]CRS-1601:CSSD Reconfiguration complete. Active nodes are crs102node1 crs102node2 .
2010-05-26 15:44:44.294
[crsd(10464)]CRS-1005:The OCR upgrade was completed. Version has changed from 169870592 to 169870592. 
Details in /u01/crs/oracle/product/10.2.0/crs/log/crs102node2/crsd/crsd.log.
2010-05-26 15:44:44.295
[crsd(10464)]CRS-1012:The OCR service started on node crs102node2.
2010-05-26 15:44:45.074
[evmd(10233)]CRS-1401:EVMD started on node crs102node2.
2010-05-26 15:44:46.452
[crsd(10464)]CRS-1201:CRSD started on node crs102node2.

The cluster is up on both nodes now, getting ready for the OCR maintanence:

[root@crs102node1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

[root@crs102node2 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

2) Oracle Cluster Registry

We first need to add an OCR mirror. /dev/xvdc9 is still “free”. It turned out in empirical testing that the OCR and mirror must be of the same partition size! I split the 5G /dev/xvdc into 5 partitions with 1024M each.

[root@crs102node1 bin]# ./ocrconfig -replace ocrmirror /dev/xvdc9

Check if that was ok:

[root@crs102node1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
 Version                  :          2
 Total space (kbytes)     :    1003728
 Used space (kbytes)      :       2016
 Available space (kbytes) :    1001712
 ID                       : 1189351742
 Device/File Name         : /dev/raw/raw1
 Device/File integrity check succeeded
 Device/File Name         : /dev/xvdc9
 Device/File integrity check succeeded

 Cluster registry integrity check succeeded

Good. Oracle actually is clever enough to update the ocr.loc file with the new information across the cluster:

[root@crs102node1 bin]# cat /etc/oracle/ocr.loc
#Device/file  getting replaced by device /dev/xvdc9 
ocrconfig_loc=/dev/raw/raw1
ocrmirrorconfig_loc=/dev/xvdc9

Now we use the newly created mirror to replace the primary copy of the OCR (this is why the partitions need to be of the same size). Note that I am not unbinding raw1 yet-you can’t as it is in use.

[root@crs102node1 bin]#./ocrconfig -replace ocr /dev/xvdc5

[root@crs102node1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
 Version                  :          2
 Total space (kbytes)     :    1003728
 Used space (kbytes)      :       2016
 Available space (kbytes) :    1001712
 ID                       : 1189351742
 Device/File Name         : /dev/xvdc5
 Device/File integrity check succeeded
 Device/File Name         : /dev/xvdc9
 Device/File integrity check succeeded

 Cluster registry integrity check succeeded

Success, too! This is also reflected in the alert_hostname.log file:

2010-05-26 15:46:16.305
[crsd(22272)]CRS-1007:The OCR/OCR mirror location was replaced by /dev/xvdc9.
2010-05-26 15:49:10.018
[crsd(22272)]CRS-1007:The OCR/OCR mirror location was replaced by /dev/xvdc5.

A quick check to really make sure everything is ok using cluvfy.

[oracle@crs102node1 bin]$ ./cluvfy comp ocr -n crs102node1,crs102node2

Verifying OCR integrity 

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...
OCR of correct Version "2" exists.

Checking data integrity of OCR...
Data integrity check for OCR passed.

OCR integrity check passed.

Verification of OCR integrity was successful. 

This concludes this exercise. I another post I’ll upgrade this to 11.2 and back to 10.2 so stay tuned.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: