Anyone to suggest a shorter title for this: you are welcome to comment! Initially I wanted to have this in the first of today’s posts, but got side tracked. So here’s the next step in the CRS 10.2.0.5 installation: moving off the deprecated raw devices to block devices. This is something you can do with OUI in 11.1, but in 10.2 it either requires raw devices or a clustered file system. Here’s the step by step protocol of my session.
The Setup
Before talking about what I am doing a little background information is in order. This example uses single pathing, the difference to multipathing is marginal-just remember to use the logical device names instead of the physical ones. In device mapper multipathing this might be similar to /dev/mapper/ocrvotep{5,6,7,8,9} in case you map your WWID for the OCR and voting disks to a logical name called “ocrvote”. If presenting only 1 device to the server to store all copies of OCR and voting disk make sure to have it well protected on the array! With this said, here is my setup. First for the voting disk-it went to /dev/raw/raw2 during the installation:
[root@crs102node1 bin]# ./crsctl query css votedisk
0. 0 /dev/raw/raw2
located 1 votedisk(s).
My OCR doesn’t have a mirror and it on /dev/raw/raw1
[root@crs102node1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 1003728
Used space (kbytes) : 2008
Available space (kbytes) : 1001720
ID : 1189351742
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
Now let’s change directory to $ORA_CRS_HOME/bin before the action starts.
Voting Disks to Block Device
[root@crs102node1 ~]# cd /u01/crs/oracle/product/10.2.0/crs/bin
To maintain voting disks it has always been a good idea to stop the cluster on ALL nodes.
[root@crs102node1 bin]# ./crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@crs102node1 bin]# ./crsctl check crs Failure 1 contacting CSS daemon Cannot communicate with CRS Cannot communicate with EVM
Ok, that’s definitely down. I repeated this on the second node:
[root@crs102node2 ~]# cd /u01/crs/oracle/product/10.2.0/crs/bin [root@crs102node2 bin]# ./crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@crs102node2 bin]# ./crsctl check crs Failure 1 contacting CSS daemon Cannot communicate with CRS Cannot communicate with EVM
I had extra raw devices mapped (raw{3,4,5}) , need to get rid of them first. Note that no restart is needed for this!
[root@crs102node1 bin]# raw -qa /dev/raw/raw1: bound to major 202, minor 37 /dev/raw/raw2: bound to major 202, minor 38 /dev/raw/raw3: bound to major 202, minor 39 /dev/raw/raw4: bound to major 202, minor 40 /dev/raw/raw5: bound to major 202, minor 41
I didn’t use udev for the raw device mapping since I knew they wouldn’t stay past a reboot. I opted for the quick and dirty /etc/rc.local method as in this example:
cat /etc/rc.local # raw devices raw /dev/raw/raw1 /dev/xvdc5 raw /dev/raw/raw2 /dev/xvdc6 raw /dev/raw/raw3 /dev/xvdc7 raw /dev/raw/raw4 /dev/xvdc8 raw /dev/raw/raw5 /dev/xvdc9 chown oracle:dba /dev/raw/raw* chmod 664 /dev/raw/raw*
By assigning raw devices to major 0/minor 0 we unbind them as the following example demonstrates:
[root@crs102node1 bin]# raw /dev/raw/raw3 0 0 /dev/raw/raw3: bound to major 0, minor 0 [root@crs102node1 bin]# raw /dev/raw/raw4 0 0 /dev/raw/raw4: bound to major 0, minor 0 [root@crs102node1 bin]# raw /dev/raw/raw5 0 0 /dev/raw/raw5: bound to major 0, minor 0
Only the required two are left. I did the same on the second node.
[root@crs102node1 bin]# raw -qa /dev/raw/raw1: bound to major 202, minor 37 /dev/raw/raw2: bound to major 202, minor 38
This is my disk layout:
[root@crs102node1 bin]# fdisk -l /dev/xvdc Disk /dev/xvdc: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/xvdc1 1 652 5237158+ 5 Extended /dev/xvdc5 1 125 1003999+ 83 Linux /dev/xvdc6 126 250 1004031 83 Linux /dev/xvdc7 251 375 1004031 83 Linux /dev/xvdc8 376 500 1004031 83 Linux /dev/xvdc9 501 625 1004031 83 Linux
The current mapping is:
/dev/raw/raw1 -> /dev/xvdc5 /dev/raw/raw2 -> /dev/xvdc6
This means that I can make use of /dev/xvdc{7,8,9} for some juggling around. Let’s get started. The plan is to add vote disk for /dev/xvdc{6,7} to have 3 voting disks in total. Remember that we need an odd number of voting disks!
[root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc6 Cluster is not in a ready state for online disk addition
Ooops, but luckily there is a force option:
[root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc6 -force Now formatting voting disk: /dev/xvdc6 successful addition of votedisk /dev/xvdc6. [root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc7 -force Now formatting voting disk: /dev/xvdc7 successful addition of votedisk /dev/xvdc7. [root@crs102node1 bin]# ./crsctl query css votedisk 0. 0 /dev/raw/raw2 1. 0 /dev/xvdc6 2. 0 /dev/xvdc7 located 3 votedisk(s).
This makes it 3 votedisks, but still one on a raw device. Let’s solve this-first unbind the raw device, then add yet another vote disk and remove the raw device from the configuration:
[root@crs102node1 bin]# raw /dev/raw/raw2 0 0 /dev/raw/raw2: bound to major 0, minor 0 [root@crs102node1 bin]# ./crsctl add css votedisk /dev/xvdc8 -force Now formatting voting disk: /dev/xvdc8 successful addition of votedisk /dev/xvdc8. [root@crs102node1 bin]# ./crsctl delete css votedisk /dev/raw/raw2 -force successful deletion of votedisk /dev/raw/raw2.
We are almost there, let’s check:
[root@crs102node1 bin]# ./crsctl query css votedisk 0. 0 /dev/xvdc6 1. 0 /dev/xvdc7 2. 0 /dev/xvdc8 located 3 votedisk(s).
Brilliant. Get ready to start the cluster-the OCR maintenance can be done while it’s online. First we need to set the permissions correctly; I added this to /etc/rc.local. The more elegant way is to use udev for this.
[root@crs102node1 bin]# chown oracle:dba /dev/xvdc{5,6,7,8,9}
[root@crs102node1 bin]# chmod 664 /dev/xvdc{5,6,7,8,9}
[root@crs102node1 bin]# ls -l /dev/xvdc{5,6,7,8,9}
brw-rw-r-- 1 oracle dba 202, 37 May 25 16:25 /dev/xvdc5
brw-rw-r-- 1 oracle dba 202, 38 May 26 15:39 /dev/xvdc6
brw-rw-r-- 1 oracle dba 202, 39 May 26 15:39 /dev/xvdc7
brw-rw-r-- 1 oracle dba 202, 40 May 26 15:41 /dev/xvdc8
brw-rw-r-- 1 oracle dba 202, 41 May 25 16:25 /dev/xvdc9
Repeat on the other node:
[root@crs102node2 bin]# chown oracle:dba /dev/xvdc{5,6,7,8,9}
[root@crs102node2 bin]# chmod 664 /dev/xvdc{5,6,7,8,9}
[root@crs102node2 bin]# ls -l /dev/xvdc{5,6,7,8,9}
brw-rw-r-- 1 oracle dba 202, 37 May 25 16:26 /dev/xvdc5
brw-rw-r-- 1 oracle dba 202, 38 May 25 16:26 /dev/xvdc6
brw-rw-r-- 1 oracle dba 202, 39 May 25 16:26 /dev/xvdc7
brw-rw-r-- 1 oracle dba 202, 40 May 25 16:26 /dev/xvdc8
brw-rw-r-- 1 oracle dba 202, 41 May 25 16:26 /dev/xvdc9
[root@crs102node2 bin]# raw /dev/raw/raw2 0 0
/dev/raw/raw2: bound to major 0, minor 0
Now start the cluster on both nodes
[root@crs102node1 bin]# ./crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@crs102node2 bin]# ./crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly Additional information about success is in $ORA_CRS_HOME/log/hostname/alert_hostname.log: 2010-05-26 15:29:58.556 [cssd(21486)]CRS-1603:CSSD on node crs102node2 shutdown by user. 2010-05-26 15:44:38.144 [cssd(10993)]CRS-1605:CSSD voting file is online: /dev/xvdc6. ... 2010-05-26 15:44:38.147 [cssd(10993)]CRS-1605:CSSD voting file is online: /dev/xvdc7. ... 2010-05-26 15:44:38.149 [cssd(10993)]CRS-1605:CSSD voting file is online: /dev/xvdc8. ... 2010-05-26 15:44:43.993 [cssd(10993)]CRS-1601:CSSD Reconfiguration complete. Active nodes are crs102node1 crs102node2 . 2010-05-26 15:44:44.294 [crsd(10464)]CRS-1005:The OCR upgrade was completed. Version has changed from 169870592 to 169870592. Details in /u01/crs/oracle/product/10.2.0/crs/log/crs102node2/crsd/crsd.log. 2010-05-26 15:44:44.295 [crsd(10464)]CRS-1012:The OCR service started on node crs102node2. 2010-05-26 15:44:45.074 [evmd(10233)]CRS-1401:EVMD started on node crs102node2. 2010-05-26 15:44:46.452 [crsd(10464)]CRS-1201:CRSD started on node crs102node2.
The cluster is up on both nodes now, getting ready for the OCR maintanence:
[root@crs102node1 bin]# ./crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy [root@crs102node2 bin]# ./crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy
2) Oracle Cluster Registry
We first need to add an OCR mirror. /dev/xvdc9 is still “free”. It turned out in empirical testing that the OCR and mirror must be of the same partition size! I split the 5G /dev/xvdc into 5 partitions with 1024M each.
[root@crs102node1 bin]# ./ocrconfig -replace ocrmirror /dev/xvdc9
Check if that was ok:
[root@crs102node1 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 1003728 Used space (kbytes) : 2016 Available space (kbytes) : 1001712 ID : 1189351742 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/xvdc9 Device/File integrity check succeeded Cluster registry integrity check succeeded
Good. Oracle actually is clever enough to update the ocr.loc file with the new information across the cluster:
[root@crs102node1 bin]# cat /etc/oracle/ocr.loc
#Device/file getting replaced by device /dev/xvdc9
ocrconfig_loc=/dev/raw/raw1
ocrmirrorconfig_loc=/dev/xvdc9
Now we use the newly created mirror to replace the primary copy of the OCR (this is why the partitions need to be of the same size). Note that I am not unbinding raw1 yet-you can’t as it is in use.
[root@crs102node1 bin]#./ocrconfig -replace ocr /dev/xvdc5 [root@crs102node1 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 1003728 Used space (kbytes) : 2016 Available space (kbytes) : 1001712 ID : 1189351742 Device/File Name : /dev/xvdc5 Device/File integrity check succeeded Device/File Name : /dev/xvdc9 Device/File integrity check succeeded Cluster registry integrity check succeeded
Success, too! This is also reflected in the alert_hostname.log file:
2010-05-26 15:46:16.305 [crsd(22272)]CRS-1007:The OCR/OCR mirror location was replaced by /dev/xvdc9. 2010-05-26 15:49:10.018 [crsd(22272)]CRS-1007:The OCR/OCR mirror location was replaced by /dev/xvdc5.
A quick check to really make sure everything is ok using cluvfy.
[oracle@crs102node1 bin]$ ./cluvfy comp ocr -n crs102node1,crs102node2 Verifying OCR integrity Checking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations. Uniqueness check for OCR device passed. Checking the version of OCR... OCR of correct Version "2" exists. Checking data integrity of OCR... Data integrity check for OCR passed. OCR integrity check passed. Verification of OCR integrity was successful.
This concludes this exercise. I another post I’ll upgrade this to 11.2 and back to 10.2 so stay tuned.
