Martins Blog

Trying to explain complex things in simple terms

ORA-15186 with ASMLib and device mapper on Linux

Posted by Martin Bach on September 5, 2009

Clumsy title but a problem that can slow you down. When creating ASM disk groups, you might not succeed, ASM complains with ORA-15186. Facts:

  • you are using ASMLib
  • you are using native Linux multipathing (device-mapper-multipath)
  • And you are using Linux
The first port of call should be the alert log of the ASM instance (who wouldn’t check this?). You can find lots of failed calls to ASMLib.

An example alert.log entry could be as follows:

SQL> ALTER DISKGROUP ALL MOUNT
Thu Aug 27 07:36:48 2009
Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
Thu Aug 27 07:36:48 2009
ORA-15186: ASMLIB error function = [asm_open],  error = [1],  mesg = [Operation not permitted]

This is a tricky one as /etc/init.d/oracleasm listdisks reports all your disks:

[oracle@host bdump]$ /etc/init.d/oracleasm listdisks
DATA1
DATA2
FRA1

The first thing which points at something which isn’t quite right is in v$asm_disk:

SQL>  select inst_id,HEADER_STATUS,STATE,NAME,path,LIBRARY
  2*    from gv$asm_disk;
INST HEADER_STATU PATH            LIBRARY
---- ------------ --------------- ------------------------------------------------------------
   2 FOREIGN      /dev/raw/raw2   System
   2 UNKNOWN      ORCL:FRA1       ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   2 UNKNOWN      ORCL:DATA2      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   2 UNKNOWN      ORCL:DATA1      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   1 FOREIGN      /dev/raw/raw2   System
   1 PROVISIONED  ORCL:FRA1       ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   1 MEMBER       ORCL:DATA1      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)
   1 MEMBER       ORCL:DATA2      ASM Library - Generic Linux, version 2.0.4 (KABI_V2)

You can see that the HEADER_STATUS on instance 2 of the ASM cluster is “UNKNOWN”, whereas it’s MEMBER on instance 1. Odd, but simple to fix. Check /etc/sysconfig/oracleasm and make sure to have SCANORDER and SCANEXCLUDE set correctly:

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm-"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

That’s it – now simply restart ASMLib and it should be able to mount the disk group.

6 Responses to “ORA-15186 with ASMLib and device mapper on Linux”

  1. franco said

    hi martin

    i have the problem with the asmlib. i have installed a oracle 10.2.0.4 rac cluster. the cluster service works fine. but when i want to create asm-diskgroups, all work fine, but after a reboot, i can’t mount the diskgroups. ora-15186 failure in the alertlog, and the same header status as you (unknown). i have read many docu but nothing works for me. i have installed on a RHEL 5.3 32bit / with IBM Faststorage 7000 on a IBM 445 Server ( old hardware :) ! ) so i also installed the qlogic hba2340 driver, and i work with the devicemapper. for crs i take /dev/raw/raw*! i set the permissions with the rc.local file. i test it with the oracleasm_scanorder, with the udev-rules (90-dm). asm_diskstring is set to ORCL. nothing will work. maybe you have a simple tip or trick for me. greetz from austria!

    franco

    • Martin said

      This is not too easy to diagnose from a distance… First of all I’d check if asmlib is enabled (/etc/sysconfig/oracleasm), and if so, the scanboot attribute should be set to yes as well. From what you said you are using some kind of multipathing in which case you have to set the scanorder and scanexclude variables accordingly. Metalink has an example for EMC storage (don’t know the note off hand), my example works for device mapper multipath. The idea is always the same: instead of using the /dev/sdxx device you use whatever the name of the logical device is. In device mapper multipath it’s dm-.

      With ASMLib you shouldn’t use raw devices or udev-asmlib will read the header of your devices and add the disks. /etc/init.d/oracleasm listdisks will show you the disks it detected after a “scandisks”, you can use /etc/init.d/oracleasm querydisk /path/to/asmdisk to check if it has an ASM header.

      I recommend the excellent Oracle Automatic Storage Management: Under-the-Hood and Practical Deployment Guide, it has it’s own chapter on asmlib.

      Hope that helps,

      Martin

      • franco said

        hi martin, thx for answer. i know diagnostic isn’t easy about the difference. my problem is maybe simple. but maybe i can tell u more about my system. i use the raw devices with udev for the clusterware. for asmlib i have add the disk to the /dev/mapper/*** devices. oracleasm is enabled, listdisk see all the discs, scandisk is also ok, so i think the header is ok. but i can only mount this disks, when i create it firsttime. i have test it with the scanorder, but doesn’t help. when i look at /dev/oracleasm/disks/…. devices, the settings/rights are set to oracle:oraasm. thats sound also good. so i don’t know, where i can look, where the problem is. i have no more idea. /etc/multipath is set with aliases and failover!

        greetz

        franco

      • Martin said

        Franco,

        in this case I’d check the alert log of the asm instance: what’s the output when it tries to mount the disk? Metalink should provide more information on this error. Also, my understanding is that you set the scanorder to “dm-“, not the devices in /dev/mapper.

        You might find additional useful information here: http://www.oracle.com/technology/tech/linux/asmlib/multipath.html

      • franco said

        hello martin

        i have now a solution, but i don’t know if that was good. i have now in the /etc/sysconfig/oracleasm the “ORACLEASM_SCANORDER=”mapper”, “ORACLEASM_SCANEXCLUDE=”sd dm”, and in the /oracle/admin/+ASM/pfile/init.ora the “asm_diskstring=’/dev/mapper/datap1′,’/dev/mapper/archivep1’……

        and now i can mount the disks. but i see the that the path is not ORCL:… and the Library is “System” not “ASM-Library….”. with dev mapper it works, but with ORCL not. i test it with the scanorder dm, and the diskstring orcl:*, than it doesn’t mount the disks. maybe asmlib 2.04 is not the best asmlib???

        greetz

        franco

  2. redvoid said

    Thanks for this Martin. This saved me today. I had run into this same issue with a RAC build about 6 months ago, but forgot the specifics, and your article brought it all right back in a second, which is great since I beat my head against the wall for 2 days last time around. :)

    FWIW: I tend to use the full device path, but I have noticed that wildcard use in this config file is not as robust as wildcard use in Linux general. By that, I mean when I try to lock it down as tight as possible with something like:

    # ORACLEASM_SCANORDER: Matching patterns to order disk scanning
    ORACLEASM_SCANORDER=”/dev/mapper/mpath-app??p1″

    I still get other paths that do not strictly fit the wildcard like:

    /dev/mapper/mpath-app01
    /dev/mapper/mpath-app01p1

    and the first one should not equate to what I specified, but it does. The ones without the partition show up as “candidate” and the ones with the “p1” partition designation show up as “provisioned” when creating the initial ASM disk groups, so I have to manually select just the provisioned ones, and ignore the candidates. No biggie, but kinda frustrating when it doesn’t behave like it looks like it should.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: