Build your own 11.2.0.2 stretched RAC part IV

Finally I have some more time to work on the next article in this series, dealing with the setup of my two cluster nodes. This is actually going to be quite short compared to the other articles so far. This is mainly due to the fact that I have streamlined the deployment of new Oracle-capable machines to a degree where I can comfortably set up a cluster in 2 hours. It’s a bit more work initially, but it paid off. The setup of my reference VM is documented on this blog as well, search for virtualisation and opensuse to get to the article.

When I first started working in my lab environment I created a virtual machine called “rhel55ref”. In reality it’s OEL, because of Red Hat’s windooze like policy to require an activation code. I would have considered CentOS as well, but when I created the reference VM the community hadn’t provided the “update 5”. I like the brand new shiny things most :)

Seems like I’m lucky now as well with the introduction of Oracle’s own Linux kernel I am ready for the future. Hopefully Red Hat will get their act together soon and release version 6 of their distribution. As much as I like Oracle I don’t want them to dominate the OS market too much. With Solaris now in their hands as well…

Anyway, to get started with my first node I cloned my template. Moving to /var/lib/xen/images all I had to do was to “cp -a rhel55ref edcnode1”. One repetition to edcnode2 gave me my second node. Xen (or libvirt for that matter) stores the VM configuration in xenstore, a backend database which can be interrogated easily. So I dumped the XML configuration file for my rhel55ref VM and stored it in edcnode{1,2}.xml. The command to dump the information is “virsh dumpxml domainName” > edcnode{1,2}.xml

The domU folder contains the virtual disk for the root file system of my VM, called disk0. I then created a new “hard disk”, called disk1 to contain the Oracle binaries. Experience told me not to have that too small, 20G should be enough for my /u01 mountpoint for Grid Infrastructure and the RDBMS binaries.

[root@dom0]# /var/lib/xen/images/edcnode1 # dd if=/dev/zero of=disk01 bs=1 count=0 seek=20G
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.3869e-05 s, 0.0 kB/

I like to speed the file creation up by using the sparse file trick: the file disk1 will be reported to be 20G in size, but it will only use that if the virtual machine needs them. It’s a bit like Oracle creating a temporary tablespace.

With that information it’s time to modify the dumped XML file. Again it’s important to define MAC addresses for the network interfaces, otherwise the system will try and use dhcp for your NICs, destroying the carefully crafted /etc/sysconfig/network-scripts/ifcfg-eth{0,1,2} files. Oh, and remember that the first 3 tupel are reserved for XEN, so don’t change “00:16:3e”! Your UUID also has to be unique. In the end my first VM’s XML description looked like this:

<domain type='xen'>
 <name>edcnode1</name>
 <uuid>46a36f98-4e52-45a5-2579-80811b38a3ab</uuid>
 <memory>4194304</memory>
 <currentMemory>524288</currentMemory>
 <vcpu>2</vcpu>
 <bootloader>/usr/bin/pygrub</bootloader>
 <bootloader_args>-q</bootloader_args>
 <os>
 <type>linux</type>
 <cmdline> </cmdline>
 </os>
 <clock offset='utc'/>
 <on_poweroff>destroy</on_poweroff>
 <on_reboot>restart</on_reboot>
 <on_crash>destroy</on_crash>
 <devices>
 <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
 <disk type='file' device='disk'>
 <driver name='file'/>
 <source file='/var/lib/xen/images/edcnode1/disk0'/>
 <target dev='xvda' bus='xen'/>
 </disk>
 <disk type='file' device='disk'>
 <driver name='file'/>
 <source file='/var/lib/xen/images/edcnode1/disk1'/>
 <target dev='xvdb' bus='xen'/>
 </disk>
 <interface type='bridge'>
 <mac address='00:16:3e:ab:cd:ef'/>
 <source bridge='br1'/>
 <script path='/etc/xen/scripts/vif-bridge'/>
 <target dev='vif-1.0'/>
 </interface>
<interface type='bridge'>
 <mac address='00:16:3e:10:13:1a'/>
 <source bridge='br2'/>
 <script path='/etc/xen/scripts/vif-bridge'/>
 </interface>
<interface type='bridge'>
 <mac address='00:16:3e:11:12:ef'/>
 <source bridge='br3'/>
 <script path='/etc/xen/scripts/vif-bridge'/>
 <target dev='vif-1.0'/>
 </interface>
 <console type='pty'>
 <target port='0'/>
 </console>
 <input type='mouse' bus='xen'/>
 </devices>
</domain>

You can see that the interfaces refer to br1, br2, and br3. These are the ones that were previously defined in the first article. The tag “<target>” in the <interface> tag doesn’t matter as that will be dynamically assigned anyway.

When done, you can define the new VM and start it:

[root@dom0]# virsh define edcnode{1,2}.xml
[root@dom0]# xm start edcnode1 -c

You are directly connected to the VM’s console (80×24-just like in the old times!) and have to wait a looooong time for the DHCP requests for eth0, eth1 and eth2 to time out. This is the first thing to address. As root, log in to the system and navigate straight to /etc/sysconfig/network-scripts to change ifcfg-eth{0,1,2}. Alternatively, use system-config-network-tui to change the network settings.

The following settings should be used for edcnode1:

• eth0:    192.168.99.56/24
• eth1:    192.168.100.56/24
• eth2:    192.168.101.56/24

These are the settings for edcnode2:

• eth0:    192.168.99.58/24
• eth1:    192.168.100.58/24
• eth2:    192.168.101.58/24

The nameserver for both is my dom0 – in this case 192.168.99.10. Enter the appropriate hostname as well as the nameserver. Note that 192.168.99.57 and 59 are reserved for the node VIPs, hence the “gap”. Then edit /etc/hosts to enter the information about the private interconnect, which for obvious reasons is not included in DNS. If you like, persist your public and VIP information in /etc/hosts as well. Don’t do this with the SCAN, it’s not suggested to have the SCAN resolve through /etc/hosts although it works.

Now’s the big moment-restart the network services and get out of the uncomfortable 80×24 character limitation:

[root@edcnode1]# service network restart

The complete configuration is printed here for the sake of completeness for edcnode1:

[root@edcnode1 ~]# cat /etc/resolv.conf
nameserver 192.168.99.10
search localdomain
[root@edcnode1 ~]#

[root@edcnode1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth{0,1,2}
# Xen Virtual Ethernet
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
HWADDR=00:16:3e:ab:cd:ef
NETMASK=255.255.255.0
IPADDR=192.168.99.56
TYPE=Ethernet
# Xen Virtual Ethernet
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
HWADDR=00:16:3e:10:13:1a
NETMASK=255.255.255.0
IPADDR=192.168.100.56
TYPE=Ethernet
# Xen Virtual Ethernet
DEVICE=eth2
BOOTPROTO=none
ONBOOT=yes
HWADDR=00:16:3e:11:12:ef
NETMASK=255.255.255.0
IPADDR=192.168.101.56
TYPE=Ethernet

[root@edcnode1 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=edcnode1

Next on the agenda is the iscsi-initiator. This isn’t part of my standard build and had to be added. All my software is exported from the dom0 via NFS and mounted to /mnt/

[root@edcnode1 ~]# find /mnt -iname "iscsi*"
/mnt/oracleEnterpriseLinux/source/iscsi-initiator-utils-6.2.0.871-0.16.el5.x86_64.rpm

[root@edcnode1 ~]# cd /mnt/oracleEnterpriseLinux/source/
[root@edcnode1 ~]# rpm -ihv iscsi-initiator-utils-6.2.0.871-0.16.el5.x86_64.rpm
warning: iscsi-initiator-utils-6.2.0.871-0.16.el5.x86_64.rpm: ...
Preparing...                ########################################### [100%]
 1:iscsi-initiator-utils  ########################################### [100%

It’s important to edit the initiator name, i.e. the name the initiator reports back to OpenFiler. I changed it to include edcnode1 and edcnode2 on their respective hosts. The file to edit is /etc/iscsi/initiatorname.iscsi

Time to get serious now:

[root@edcnode1 ~]# /etc/init.d/iscsi start
iscsid is stopped
Starting iSCSI daemon:                                     [  OK  ]
 [  OK  ]
Setting up iSCSI targets: iscsiadm: No records found!
 [  OK  ]

We are ready to roll. First, we need to discover the targets from the OpenFiler appliance-start with the first one filer01:

[root@edcnode1 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.101.50
192.168.101.50:3260,1 iqn.2006-01.com.openfiler:asm01Filer01
192.168.101.50:3260,1 iqn.2006-01.com.openfiler:ocrvoteFiler01
192.168.101.50:3260,1 iqn.2006-01.com.openfiler:asm02Filer01

A restart of the iscsi service will automatically log in and persist the settings (this is very wide output-works best in 1280xsomething resolution)

[root@edcnode1 ~]# service iscsi restart
Stopping iSCSI daemon:
iscsid dead but pid file exists                            [  OK  ]
Starting iSCSI daemon:                                     [  OK  ]
 [  OK  ]
Setting up iSCSI targets: Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm01Filer01, portal: 192.168.101.50,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:ocrvoteFiler01, portal: 192.168.101.50,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm02Filer01, portal: 192.168.101.50,3260]
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm01Filer01, portal: 192.168.101.50,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:ocrvoteFiler01, portal: 192.168.101.50,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm02Filer01, portal: 192.168.101.50,3260]: successful
 [  OK  ]

Fine! Now over to fdisk the new devices. I know that my “local” storage is named /dev/xvd*, so anything new (“/dev/sd*”) will be iSCSI provided storage. If you are unsure you can always check the /var/log/messages file to see which device have just been discovered. You should see something similar to this output:

Sep 24 12:20:08 edcnode1 kernel: Loading iSCSI transport class v2.0-871.
Sep 24 12:20:08 edcnode1 kernel: cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
Sep 24 12:20:08 edcnode1 kernel: iscsi: registered transport (cxgb3i)
Sep 24 12:20:08 edcnode1 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.1.0 (Oct 10, 2009)
Sep 24 12:20:08 edcnode1 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.0 (Dec 06, 2009)
Sep 24 12:20:08 edcnode1 kernel: iscsi: registered transport (bnx2i)
Sep 24 12:20:08 edcnode1 kernel: iscsi: registered transport (tcp)
Sep 24 12:20:08 edcnode1 kernel: iscsi: registered transport (iser)
Sep 24 12:20:08 edcnode1 kernel: iscsi: registered transport (be2iscsi)
Sep 24 12:20:08 edcnode1 iscsid: iSCSI logger with pid=20558 started!
Sep 24 12:20:08 edcnode1 kernel: scsi0 : iSCSI Initiator over TCP/IP
Sep 24 12:20:08 edcnode1 kernel: scsi1 : iSCSI Initiator over TCP/IP
Sep 24 12:20:08 edcnode1 kernel: scsi2 : iSCSI Initiator over TCP/IP
Sep 24 12:20:09 edcnode1 kernel:   Vendor: OPNFILER  Model: VIRTUAL-DISK      Rev: 0
Sep 24 12:20:09 edcnode1 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
Sep 24 12:20:09 edcnode1 kernel:   Vendor: OPNFILER  Model: VIRTUAL-DISK      Rev: 0
Sep 24 12:20:09 edcnode1 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
Sep 24 12:20:09 edcnode1 kernel:   Vendor: OPNFILER  Model: VIRTUAL-DISK      Rev: 0
Sep 24 12:20:09 edcnode1 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
Sep 24 12:20:09 edcnode1 kernel:   Vendor: OPNFILER  Model: VIRTUAL-DISK      Rev: 0
Sep 24 12:20:09 edcnode1 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
Sep 24 12:20:09 edcnode1 kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 0
Sep 24 12:20:09 edcnode1 kernel: scsi 1:0:0:0: Attached scsi generic sg1 type 0
Sep 24 12:20:09 edcnode1 kernel: scsi 2:0:0:0: Attached scsi generic sg2 type 0
Sep 24 12:20:09 edcnode1 kernel: scsi 1:0:0:1: Attached scsi generic sg3 type 0
Sep 24 12:20:09 edcnode1 kernel: SCSI device sda: 20971520 512-byte hdwr sectors (10737 MB)
Sep 24 12:20:09 edcnode1 kernel: sda: Write Protect is off
Sep 24 12:20:09 edcnode1 kernel: SCSI device sda: drive cache: write through
Sep 24 12:20:09 edcnode1 kernel: SCSI device sda: 20971520 512-byte hdwr sectors (10737 MB)
Sep 24 12:20:09 edcnode1 kernel: sda: Write Protect is off
Sep 24 12:20:09 edcnode1 kernel: SCSI device sda: drive cache: write through
Sep 24 12:20:09 edcnode1 kernel:  sda: unknown partition table
Sep 24 12:20:09 edcnode1 kernel: sd 0:0:0:0: Attached scsi disk sda

The output will continue with /dev/sdb and other devices exported by the filer.

Prepare the local Oracle Installation

Using fdisk, modify /dev/xvdb, create a partition spanning the whole disk and set its type to “8e” – Linux LVM. It’s always a good idea to use LVM to install Oracle binaries into, it makes later extension of a filesystem easier. I’ll add the fdisk output here for this device but won’t for later partitioning excercises.

root@edcnode1 ~]# fdisk /dev/xvdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
 (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
 e   extended
 p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):
Using default value 1305

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Once /dev/xvdb1 is ready, we need to start its transformation into a logical volume. First, a physical volume is to be created:

[root@edcnode1 ~]# pvcreate /dev/xvdb1
 Physical volume "/dev/xvdb1" successfully created

The physical volume (“PV”) is then used to form a volume group (“VG”). In real life, you’d probably have more than 1 PV to form a VG… I named my volume group “oracle_vg”. The existing volume group is called “root_vg” by the way.

[root@edcnode1 ~]# vgcreate oracle_vg /dev/xvdb1
 Volume group "oracle_vg" successfully create

Wonderful! I never quite remember how many extents this VG has so I need to query it. When using –size 10g it will through an error – some internal overhead will reduce the available capacity to something just shy of 10G:

[root@edcnode1 ~]# vgdisplay oracle_vg
 --- Volume group ---
 VG Name               oracle_vg
 System ID
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  1
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                0
 Open LV               0
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               10.00 GB
 PE Size               4.00 MB
 Total PE              2559
 Alloc PE / Size       0 / 0
 Free  PE / Size       2559 / 10.00 GB
 VG UUID               QgHgnY-Kqsl-noAR-VLgP-UXcm-WADN-VdiwO7

Right, so now let’s create a logical volume (“LV”) with 2559 extents:

[root@edcnode1 ~]# lvcreate --extents 2559 --name grid_lv oracle_vg
 Logical volume "grid_lv" created

And now we need a file system:

[root@edcnode1 ~]# mkfs.ext3 /dev/oracle_vg/grid_lv

You are done! Create the mountpoint for your oracle installation, /u01/ in my case, and grant oracle:oinstall ownership to it. In this lab excercise I didn’t create a separate owner for the Grid Infrastructure to avoid potentially undiscovered problems in 11.2.0.2 and stretched RAC. Finally add this to /etc/fstab to make it persistent:

[root@edcnode1 ~]# echo "/dev/oracle_vg/grid_lv   /u01   ext3 defaults 0 0" >> /etc/fstab
[root@edcnode1 ~]# mount /u01

Now continue to partition the iSCSI volumes, but don’t create file systems on top of them. You should not assign a partition type other than the default “Linux” to it either.

ASMLib

Yes I know…The age old argument, but I decided to use it anyway. The reason is simple: scsi_id doesn’t return a value in para-virtualised Linux, which makes it impossible to set up device name persistence with udev. And ASMLib is easier to use anyway! But if your system administrators are database agnostic and not willing to learn the basics about ASM, then probably ASMLib is not a good idea to be rolled out. It’s only a matter of time until someone executes an “rpm -Uhv kernel*” to your box and of course a) didn’t tell the DBAs and b) didn’t bother applying the ASMLib kernel module. But I digress.

Before you are able to use ASMLib you have to configure it on each cluster node. A sample session could look like this:

[root@edcnode1 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@edcnode1 ~]#

Now with this done, it is possible to create the ASMLib maintained ASM disks. For the LUNs presented by filer01 these be

• ASM01FILER01
• ASM02FILER01
• OCR01FILER01
• OCR02FILER01

The disks are created using the /etc/init.d/oracleasm createdisk command as in these examples:

[root@edcnode1 ~]# /etc/init.d/oracleasm createdisk asm01filer01 /dev/sda1
Marking disk "asm01filer01" as an ASM disk:                [  OK  ]
[root@edcnode1 ~]# /etc/init.d/oracleasm createdisk asm02filer01 /dev/sdc1
Marking disk "asm02filer01" as an ASM disk:                [  OK  ]
[root@edcnode1 ~]# /etc/init.d/oracleasm createdisk ocr01filer01 /dev/sdb1
Marking disk "ocr01filer01" as an ASM disk:                [  OK  ]
[root@edcnode1 ~]# /etc/init.d/oracleasm createdisk ocr02filer01 /dev/sdd1
Marking disk "ocr02filer01" as an ASM disk:                [  OK  ]

Switch over to the second node now to validate the configuration and to continue the configuration of the iSCSI LUNs from filer02. Define the domU with a similar configuration file as shown above for edcnode1, and start the domU. Once the wait for DHCP timeouts is over and you are presented with a login, set up the network as shown above. Install the iscsi initiator package, change the initiator name and discover the targets from filer02 in addition to those from filer01.

[root@edcnode2 ~]# iscsiadm -t st -p 192.168.101.51 -m discovery
192.168.101.51:3260,1 iqn.2006-01.com.openfiler:asm02Filer02
192.168.101.51:3260,1 iqn.2006-01.com.openfiler:ocrvoteFiler02
192.168.101.51:3260,1 iqn.2006-01.com.openfiler:asm01Filer02
[root@edcnode2 ~]# iscsiadm -t st -p 192.168.101.50 -m discovery
192.168.101.50:3260,1 iqn.2006-01.com.openfiler:asm01Filer01
192.168.101.50:3260,1 iqn.2006-01.com.openfiler:ocrvoteFiler01
192.168.101.50:3260,1 iqn.2006-01.com.openfiler:asm02Filer01

Still on the second node, continue the mounting of the scsi devices

[root@edcnode2 ~]# service iscsi start
iscsid (pid  2802) is running...
Setting up iSCSI targets: Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm02Filer02, portal: 192.168.101.51,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:ocrvoteFiler02, portal: 192.168.101.51,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm01Filer01, portal: 192.168.101.50,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm01Filer02, portal: 192.168.101.51,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:ocrvoteFiler01, portal: 192.168.101.50,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm02Filer01, portal: 192.168.101.50,3260]
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm02Filer02, portal: 192.168.101.51,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:ocrvoteFiler02, portal: 192.168.101.51,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm01Filer01, portal: 192.168.101.50,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm01Filer02, portal: 192.168.101.51,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:ocrvoteFiler01, portal: 192.168.101.50,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm02Filer01, portal: 192.168.101.50,3260]: successful

Partition the disks from filer02 the same way as shown in the previous example. On edcnode2, fdisk reported the following as new disks

Disk /dev/sda doesn't contain a valid partition table
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdf doesn't contain a valid partition table

Disk /dev/sda: 10.6 GB, 10670309376 bytes
Disk /dev/sdb: 2650 MB, 2650800128 bytes
Disk /dev/sdf: 10.7 GB, 10737418240 bytes

Note that /dev/sda and /dev/sdf are the 2 10G LUNs for ASM data, and /dev/sdb is the OCR/voting disk combination. Next, create the additional ASMLib disks:

[root@edcnode2 ~]# /etc/init.d/oracleasm scandisks
...
[root@edcnode2 ~]# /etc/init.d/oracleasm createdisk asm01filer02 /dev/sda1
Marking disk "asm01filer02" as an ASM disk:                [  OK  ]
[root@edcnode2 ~]# /etc/init.d/oracleasm createdisk asm02filer02 /dev/sdf1
Marking disk "asm02filer02" as an ASM disk:                [  OK  ]
[root@edcnode2 ~]# /etc/init.d/oracleasm createdisk ocr01filer02 /dev/sdb1
Marking disk "ocr01filer02" as an ASM disk:                [  OK  ]
[root@edcnode2 ~]# /etc/init.d/oracleasm listdisks
ASM01FILER01
ASM01FILER02
ASM02FILER01
ASM02FILER02
OCR01FILER01
OCR01FILER02
OCR02FILER01

Perform another scandisks command on edcnode1 to have all the disks:

[root@edcnode1 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@edcnode1 ~]# /etc/init.d/oracleasm listdisks
ASM01FILER01
ASM01FILER02
ASM02FILER01
ASM02FILER02
OCR01FILER01
OCR01FILER02
OCR02FILER01

Summary

All done!And I seriously thought initially that this was going to be a shorter post than the others, how wrong I was. Congratulations on having arrived here at the bottom of the article by the way.

In the course of this post I prepared my virtual machines to begin the installation of Grid Infrastructure. The ASM disk names will be persistent across reboots thanks to ASMLib, and no messing around with udev for that matter. You might notice that there are 2 ASM disk from filer01 but only 1 from filer02 for the voting disk/OCR diskgroup, and that’s for a reason. I’m cheeky and won’t tell you here, that’s for another post later…