Let there be multipathing (when there wasn’t before)

One of the projects I was working on recently couldn’t be finished as it should have been due to a lack of switch ports in the SAN. The QA database servers therefore only had single pathing to the storage whereas the production kit (thank god!) had multipathing enabled and configured correctly.

UPDATE 221219: this document refers to software that has since reached its End Of Life date. This article is now archived and shouldn’t be referred to.

This was far from ideal but couldn’t be helped, and it was also out of my control. You could argue that the project managers could have been aware of this and factor it in but I digress. On top of that I had a strange situation where the 3 nodes mapped the LUNs differently-what was seen as /dev/sda on node 1was /dev/sdf on the second node (but /dev/sda again on the third). The presentation of the LUNs to the database servers on the array was identical for each of the 3 nodes. How did I find out? I asked the storage admin to provide the WWIDs as seen on the management console and compared these against /dev/disk/by-id/. This only works with single pathing as explained further down.

The Linux kernel documentation states that the enumeration of devices isn’t static, and unfortunately there is no direct link between controller, target, disk and slice as in Solaris. These are RHEL 5.3 64bit system by the way, and Oracle 10.2.0.4.1 RAC.

There are 2 ways to get around this problem:

• Metalink note 414897.1 describes how to set up udev for this purpose (and it includes 11.2!)
• Use /dev/disk/by-id and /etc/rc.local

Advertisement

ORA-15186 with ASMLib and device mapper on Linux

Clumsy title but a problem that can slow you down. When creating ASM disk groups, you might not succeed, ASM complains with ORA-15186.

UPDATE 221106: Oracle Linux 5/RedHat 5 and Oracle 11.2 are effectively out of support, this article is now archived and should not be referred to. ASM Filter Driver supersedes ASMLib, I have written about ASM Filter Driver as well

Facts:

• you are using ASMLib
• you are using native Linux multipathing (device-mapper-multipath)
• And you are on RedHat/Oracle Linux 5.x
• Oracle 11.2 Real Application Clusters

The first port of call should be the alert log of the ASM instance (who wouldn’t check this?). You can find lots of failed calls to ASMLib.

An example alert.log entry could be as follows:

SQL> ALTER DISKGROUP ALL MOUNT
Thu Aug 27 07:36:48 2009
Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
Thu Aug 27 07:36:48 2009
ORA-15186: ASMLIB error function = [asm_open],  error = [1],  mesg = [Operation not permitted]

This is a tricky one as /etc/init.d/oracleasm listdisks reports all your disks:

[oracle@host bdump]$ /etc/init.d/oracleasm listdisks
DATA1
DATA2
FRA1

The first thing which points at something which isn’t quite right is in v$asm_disk:

SQL>  select inst_id,HEADER_STATUS,STATE,NAME,path,LIBRARY
  2*    from gv$asm_disk;
INST HEADER_STATU PATH            LIBRARY
---- ------------ --------------- ------------------------------------------------------------
   2 FOREIGN      /dev/raw/raw2   System
   2 UNKNOWN      ORCL:FRA1       ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   2 UNKNOWN      ORCL:DATA2      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   2 UNKNOWN      ORCL:DATA1      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   1 FOREIGN      /dev/raw/raw2   System
   1 PROVISIONED  ORCL:FRA1       ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   1 MEMBER       ORCL:DATA1      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   1 MEMBER       ORCL:DATA2      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)

You can see that the HEADER_STATUS on instance 2 of the ASM cluster is “UNKNOWN”, whereas it’s MEMBER on instance 1. Odd, but simple to fix. Check /etc/sysconfig/oracleasm and make sure to have SCANORDER and SCANEXCLUDE set correctly:

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm-"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

That’s it – now simply restart ASMLib and it should be able to mount the disk group.