New option for configuring multipathing in Oracle Linux 7

Teaching is and remains the best way for picking up new things :) While updating notes for a class I came across an interesting change in the way the device-mapper multipath works in Oracle Linux 7.4.

In the past, everyone including me used scsi_id to get the WWID of a LUN for use with dm-multipath. This is still the way Oracle documents it for Oracle Linux 7.

The utility was specified as an argument to getuid_callout in /etc/multipath.conf, as shown here:

defaults {
    getuid_callout        "/lib/udev/scsi_id --whitelisted --device=/dev/%n"

This is how it was done more or less since Red Hat 5 days. The location of scsi_id has changed over time, as expertly described on Oracle Base for example.

I don’t exactly know what happened to getuid_callout, but I can’t find it in the DM Multipath documentation for Red Hat Enterprise Linux 7. It certainly is present in its counterpart for Red Hat Enterprise Linux 6. A quick search using my favourite search engine did not reveal any clues as to what happened. The closest I got to an answer was a page in the SLES 11 SP3 documentation, which indicated that you should use a different parameter than getuid_callout.

Now I was curious! If I continued using getuid_callout, would the service stop working? Or just throw a warning?

The experiment

I quickly spun up a VM on KVM, running Oracle Linux 7.4 which I updated to the latest and greatest version of each package as of April 3rd 2018. The current versions of dm-multipath as of April 3rd 2018 are:

[root@server6 ~]# rpm -qa | grep -i multipath
[root@server6 ~]# 

I have presented a small “LUN” to the VM, using the virtual SCSI driver. I initially tried with the virtio driver but found it was not exporting necessary properties back to the VM that I need later in this post. It shouldn’t matter though, all of this is a playground and I am most interested in the concepts, not the details.

You will also notice that I am only showing snippets of the multipath.conf file. The reason for this is simple: if you don’t follow your storage vendor’s documentation detailing how to configure dm-multipath for the storage system in use, you might run into issues. Always consult your storage vendor’s official documentation!

Installing dm-multipath

I didn’t have multipathing installed initially and had to install it first. Here is the transcript:

[root@server6 ~]# yum install device-mapper-multipath


--> Running transaction check
---> Package boost-system.x86_64 0:1.53.0-27.el7 will be installed
---> Package boost-thread.x86_64 0:1.53.0-27.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

 Package                            Arch         Version                 Repository        Size
 device-mapper-multipath            x86_64       0.4.9-111.el7_4.2       ol7_latest       134 k
Installing for dependencies:
 boost-system                       x86_64       1.53.0-27.el7           ol7_latest        39 k
 boost-thread                       x86_64       1.53.0-27.el7           ol7_latest        57 k
 device-mapper-multipath-libs       x86_64       0.4.9-111.el7_4.2       ol7_latest       251 k
 librados2                          x86_64       1:0.94.5-2.el7          ol7_latest       1.7 M
Updating for dependencies:
 kpartx                             x86_64       0.4.9-111.el7_4.2       ol7_latest        72 k

Transaction Summary
Install  1 Package  (+4 Dependent packages)
Upgrade             ( 1 Dependent package)

Total download size: 2.2 M


Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : boost-system-1.53.0-27.el7.x86_64                                            1/7 
  Installing : boost-thread-1.53.0-27.el7.x86_64                                            2/7 
  Installing : 1:librados2-0.94.5-2.el7.x86_64                                              3/7 
  Installing : device-mapper-multipath-libs-0.4.9-111.el7_4.2.x86_64                        4/7 
  Updating   : kpartx-0.4.9-111.el7_4.2.x86_64                                              5/7 
  Installing : device-mapper-multipath-0.4.9-111.el7_4.2.x86_64                             6/7 
  Cleanup    : kpartx-0.4.9-111.el7.x86_64                                                  7/7 
  Verifying  : boost-system-1.53.0-27.el7.x86_64                                            1/7 
  Verifying  : 1:librados2-0.94.5-2.el7.x86_64                                              2/7 
  Verifying  : boost-thread-1.53.0-27.el7.x86_64                                            3/7 
  Verifying  : device-mapper-multipath-0.4.9-111.el7_4.2.x86_64                             4/7 
  Verifying  : kpartx-0.4.9-111.el7_4.2.x86_64                                              5/7 
  Verifying  : device-mapper-multipath-libs-0.4.9-111.el7_4.2.x86_64                        6/7 
  Verifying  : kpartx-0.4.9-111.el7.x86_64                                                  7/7 

  device-mapper-multipath.x86_64 0:0.4.9-111.el7_4.2                                            

Dependency Installed:
  boost-system.x86_64 0:1.53.0-27.el7                      boost-thread.x86_64 0:1.53.0-27.el7 
  device-mapper-multipath-libs.x86_64 0:0.4.9-111.el7_4.2  librados2.x86_64 1:0.94.5-2.el7     

Dependency Updated:
  kpartx.x86_64 0:0.4.9-111.el7_4.2                                                             

[root@server6 ~]# 

First configuration attempt

Next I quickly created a config file based on what I used in the past, making sure I add getuid_callout in the defaults {} section. And sure enough, when starting the service it complains about illegal syntax:

[root@server6 ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-04-03 14:56:19 BST; 1min 28s ago
  Process: 586 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
  Process: 568 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
  Process: 564 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 596 (multipathd)
   CGroup: /system.slice/multipathd.service
           └─596 /sbin/multipathd

Apr 03 14:56:19 server6 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Apr 03 14:56:19 server6 multipath[568]: Apr 03 14:56:19 | /etc/multipath.conf line 24, invalid keyword: getuid_callout
Apr 03 14:56:19 server6 systemd[1]: Started Device-Mapper Multipath Device Controller.
Apr 03 14:56:19 server6 multipathd[596]: /etc/multipath.conf line 24, invalid keyword: getuid_callout

Note the line reading “/etc/multipath.conf line 24, invalid keyword: getuid_callout”. It didn’t seem to prevent the meta-device from being created, but I didn’t like the error message popping up all the time.

After removing the getuid_callout from my multipath.conf file to let the defaults take over, the error went away after a restart of the service:

Apr 03 15:02:54 server6 systemd[1]: Stopped Device-Mapper Multipath Device Controller.

Apr 03 15:10:43 server6 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Apr 03 15:10:43 server6 systemd[1]: Started Device-Mapper Multipath Device Controller.
Apr 03 15:10:43 server6 multipathd[11593]: mpatha: load table [0 4194304 multipath 0 0 1 1 service-time 0 1 1 8:0 1]
Apr 03 15:10:43 server6 multipathd[11593]: mpatha: event checker started
Apr 03 15:10:43 server6 multipathd[11593]: path checkers start up

So I guess I have to rethink my approach to dm-multipath and device name persistence if I want to avoid this error message.

The New Way of doing things

I have once been told by a good friend: “when unsure, consult the documentation”. On my Linux system, my first stop is the man page. After a quick “man multipath.conf” I learned that getuid_callout is nowhere to be found. The SUSE note mentioned a new attribute, named uid_attribute, which seems to have taken its place:

The udev attribute providing a unique path identifier. Default value is ID_SERIAL

I also found this to be valuable about the “multipaths” section:

This section defines the multipath topologies. They are indexed by a World Wide Identifier (wwid), which is taken to be the value of the udev attribute given by the uid_attribute keyword.

Aha! The new model is potentially much more flexible than the old one.

Controlling aliases with multipath {} sections

I really like to assign alias names to LUNs mapped to my database systems. This way, I know that a device named “/dev/mapper/oracle_asm_data_001” is most likely used as the first ASM disk in disk group DATA. If I see /dev/mapper/mpatha instead, I know precisely nothing about its purpose.

Giving LUNs a name is not hard, you can do this in the multipaths {} section of /etc/multipath.conf. I really like doing this for reasons just mentioned. Which options do I have?

The easiest one is quite simply a change of getuid_callout to uid_attribute or its removal. Thankfully – at least in my case – the output of scsi_id and the ID_SERIAL attribute provided by udev are nearly identical:

[root@server6 ~]# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda
[root@server6 ~]# udevadm info /dev/sda | grep ID_SERIAL
E: ID_SERIAL=0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0
E: ID_SERIAL_SHORT=drive-scsi0-0-0-0
[root@server6 ~]# 

Note that I included the –replace-whitespace option in the scsi_id command, otherwise the output wouldn’t be identical. This is a concession to my lab, on real hardware I haven’t seen white space in WWIDs (this doesn’t mean there aren’t). Removing the getuid_callout from the default {} section was actually enough to create a persistent device name.

multipaths {

  multipath {
    wwid  "0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0"
    alias "oracle_asm_data_001"

And yes, I get my alias after restarting the multipathing daemon.

Apr 03 16:40:49 server6 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Apr 03 16:40:49 server6 systemd[1]: Started Device-Mapper Multipath Device Controller.
Apr 03 16:40:49 server6 multipathd[12271]: oracle_asm_data_001: load table [0 4194304 multipath 0 0 1 1 service-time 0 1 1 8:0 1]
Apr 03 16:40:49 server6 multipathd[12271]: oracle_asm_data_001: event checker started
Apr 03 16:40:49 server6 multipathd[12271]: path checkers start up

[root@server6 ~]# multipath -l
oracle_asm_data_001 (0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0) dm-3 QEMU    ,QEMU HARDDISK   
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 0:0:0:0 sda 8:0 active undef unknown
[root@server6 ~]# 

There is of course no second path, and there is a fair bit of information missing that your would otherwise get from an enterprise deployment, but you get the idea.