Martins Blog

Trying to explain complex things in simple terms

A more user friendly multipath.conf

Posted by Martin Bach on June 1, 2011

Update

The below described changes to the device uid, gid and mode have been deprecated with RHEL6/OL 6. Back to udev rules… See http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/pdf/DM_Multipath/Red_Hat_Enterprise_Linux-6-DM_Multipath-en-US.pdf for more information

During some recent work I did involving a stretched 11.2.0.2 RAC for a SAP implementation at a customer site I researched TimeFinder/Clone backups. As part of this exercise I have been able to experiment with RHEL (OEL) 5.6 and the new device mapper multipath package on the mount host. I have been very pleasantly surprise about this new feature which I’d like to share.

Background of this article

Device Mapper Multipath is the “native” Linux multipathing software, as opposed to vendor-supplied multipathing such as EMC’s Power Path or Hitachi’s HDLM.

My customer’s setup is rather unique for a SAP environment as it uses Oracle Enterprise Linux and not Solaris/SPARC or AIX on the Power platform with an active/passive solution. Well if that doesn’t make it sound unique, the fact that there is a plan to run Oracle 11.2.0.2 RAC potentially across sites using ASM and ACFS certainly makes this deployment stand out from the rest.

The reason I mention this is simple-it requires a lot of engineering effort to certify components for this combination. For example: it was not trivial to get vendor support for Solutions Enabler the storage engineering uses for connectivity with the VMAX arrays, and so on. After all, Oracle Enterprise is a fairly new platform and Red Hat certainly has an advantage when it comes to vendor certification.

What hasn’t been achieved was a certification of the EMC Power Path software for use with SAP on Linux, for reasons unknown to me. The result was simple: the setup will use the device-mapper multipath package that comes with the Linux distribution.

Configuring multipath

Now with this said I started looking at MOS to get relevant support notes about Linux’s native multipath and found some. The summary of this research is available on this blog, I have written about it in these articles:

What I didn’t know up to date was the fact that the multipath.conf file allows you to define the ownership and mode of a device. As an ASM user this is very important to me. Experience taught me that incorrect permissions are one of the main reasons for ASM failing to start. Remember that root owns block devices by default.

Consider this example:

multipaths {
...
multipath {
wwid        360a98000486e58526c34515944703277
alias       ocr01
mode        660
uid         501
gid         502
}
...
}

The above entry in the multipaths section translates into English as follows:

  • If you encounter a device with the WWID 36a…277
  • Give it an alias name of “OCR01” in /dev/mapper
  • Set the mode to 660 (i.e. rw-rw—)
  • Assign device ownership to the user with UID 501 (maps to “grid” on my system)
  • Assign the group of the device to 502 (maps to “asmdba” on my system)

The path settings are defined globally and don’t need to be mentioned explicitly for each device unless you prefer to override them. I like to use an alias although it isn’t really necessary since ASM relies on a 4k header in a block device to store its identity. If you don’t chose to alias a device I recommend you use the user friendly name instead, mainly for aesthetic reasons.

Why is this really cool? For me it implies two things:

  • I can now be independent of ASMLib which provided device name stability and set the permissions on the block devices correctly for ASM
  • I don’t need to create udev rules to set the permissions (or /etc/rc.local or whichever other way you chose before to set permissions)

Nice! One less headache, as I have to say that I didn’t really like udev…

8 Responses to “A more user friendly multipath.conf”

  1. Greg said

    Nice post .
    Btw there is little typo “mode 660” and in explanation “Set the mode to 666 (i.e. rw-rw-rw)”.
    Regards
    GregG

  2. Bernhard said

    Hi Martin,

    I would like to provide the feedback the mentioned multipath options are introduced in RHEL (OEL) 5.5 release.

    kind regards,
    Bernhard

  3. Steven Andrew said

    Hi Martin,

    This is a nice feature indeed. If this removes the need to install ASMLib, then its worth investigating it as it doesn’t seems to be as complex as udev is.

    One quick question though. We usually set the oracleasm_scanorder in oracleasm file to force asm to find dm device first and leave asm_diskstring to default. Now without ASMLib, i guess we need to include /dev/mapper/asm*, assuming all our aliases start with asm.

    • Martin Bach said

      Hi Steven,

      Thanks for your comment. In regards to your question: you are correct in your assumption. I found several references in the Red Hat documentation saying that of all dm-multipath devices you should really only use those in /dev/mapper/. It is also a good idea-like you suggest-to narrow the diskstring to /dev/mapper/asm* to speed up disk discovery if possible.

      Hope this helps!

      Martin

  4. jcnars said

    This (configuring permissions via multipath.conf) will not work in OEL 6. ASMLib should take care of it under /dev/oracleasm

    • Martin Bach said

      Hi there,

      you are right, but if you look at the update section right at the top this has already been included in the post.

      I’d be interested in your experience with ASMLib in Oracle Linux 6-have you been able to get it to work? If so, how, and with what kernel? I’m sure the other readers of the blog would be interested in your solution too. Looking forward to your feedback.

      Regards,

      Martin

      • jcnars said

        Even if you are running the latest and greatest OEL 6.x, the ASMLib kernel driver is not available any more.
        There’s no downloadable rpm of the form oracleasm-kernel-version.arch.rpm for Oracle Linux 6

        2 scenarios:
        (1) Your kernel is UEK…the driver is already in the kernel build…you just need to ensure the oracleasm mod is loaded
        (2) Your kernel is RH compatible (ex.:2.6.32-279.5.2.el6). Oracle says, “good luck folks there’s no ASMLib and you are on your own”. So, even though you are on a “oracle-on-oracle” platform (as the sales folks like to call it), you got to set the device name persistence using udev rules after doing the multipathing.

        [root@oralina ~]# oracleasm update-driver
        Kernel: 2.6.32-279.5.2.el6.x86_64 x86_64
        Driver name: oracleasm-2.6.32-279.5.2.el6.x86_64
        Driver for kernel 2.6.32-279.5.2.el6.x86_64 does not exist

        [root@oralina ~]# uname -a
        Linux oralina.KP.com 2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

        [root@oralina ~]# yum install oracleasm-`uname -r`
        Loaded plugins: rhnplugin, security
        Setting up Install Process
        No package oracleasm-2.6.32-279.5.2.el6.x86_64 available.
        Error: Nothing to do

        [root@oralina ~]# rpm -qa | grep -i oracle | sort -u
        oracleasmlib-2.0.4-1.el6.x86_64
        oracleasm-support-2.1.5-1.el6.x86_64
        oraclelinux-release-6Server-3.0.2.x86_64
        oraclelinux-release-notes-6Server-7.x86_64
        oracle-logos-60.0.11-9.el6.noarch
        oracle-rdbms-server-11gR2-preinstall-1.0-6.el6.x86_64

        [root@oralina ~]# yum list \*oracle\*
        Loaded plugins: rhnplugin, security
        Installed Packages
        oracle-logos.noarch 60.0.11-9.el6 @anaconda-OracleLinuxServer-201206261930.x86_64/6.3
        oracle-rdbms-server-11gR2-preinstall.x86_64 1.0-6.el6 @ol6_x86_64_latest
        oracleasm-support.x86_64 2.1.5-1.el6 @ol6_x86_64_latest
        oracleasmlib.x86_64 2.0.4-1.el6 @ol6_x86_64_oracle
        oraclelinux-release.x86_64 6:6Server-3.0.2 @anaconda-OracleLinuxServer-201206261930.x86_64/6.3
        oraclelinux-release-notes.x86_64 6Server-7 @anaconda-OracleLinuxServer-201206261930.x86_64/6.3
        Available Packages
        oracle-instantclient11.2-basic.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle
        oracle-instantclient11.2-basiclite.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle
        oracle-instantclient11.2-devel.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle
        oracle-instantclient11.2-jdbc.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle
        oracle-instantclient11.2-odbc.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle
        oracle-instantclient11.2-precomp.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle
        oracle-instantclient11.2-sqlplus.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle
        oracle-instantclient11.2-tools.x86_64 11.2.0.3.0-1 ol6_x86_64_oracle

        As you can see, even though we have ULN subscription, this is clearly arm-twisting by Oracle to wean customers away from RH.
        We have NetApp SMO which won’t play nice in UEK, so we are sticking with this RH-compatible kernel.

        thanks Martin…always a pleasure to learn thru’ your adventures & laze around in your blog.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: