Teaching is and remains the best way for picking up new things :) While updating notes for a class I came across an interesting change in the way the device-mapper multipath works in Oracle Linux 7.4.
In the past, everyone including me used scsi_id to get the WWID of a LUN for use with dm-multipath. This is still the way Oracle documents it for Oracle Linux 7.
The utility was specified as an argument to getuid_callout in /etc/multipath.conf, as shown here:
defaults {
[...]
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
[...]
}
This is how it was done more or less since Red Hat 5 days. The location of scsi_id has changed over time, as expertly described on Oracle Base for example.
I don’t exactly know what happened to getuid_callout, but I can’t find it in the DM Multipath documentation for Red Hat Enterprise Linux 7. It certainly is present in its counterpart for Red Hat Enterprise Linux 6. A quick search using my favourite search engine did not reveal any clues as to what happened. The closest I got to an answer was a page in the SLES 11 SP3 documentation, which indicated that you should use a different parameter than getuid_callout.
Now I was curious! If I continued using getuid_callout, would the service stop working? Or just throw a warning?
The experiment
I quickly spun up a VM on KVM, running Oracle Linux 7.4 which I updated to the latest and greatest version of each package as of April 3rd 2018. The current versions of dm-multipath as of April 3rd 2018 are:
[root@server6 ~]# rpm -qa | grep -i multipath device-mapper-multipath-0.4.9-111.el7_4.2.x86_64 device-mapper-multipath-libs-0.4.9-111.el7_4.2.x86_64 [root@server6 ~]#
I have presented a small “LUN” to the VM, using the virtual SCSI driver. I initially tried with the virtio driver but found it was not exporting necessary properties back to the VM that I need later in this post. It shouldn’t matter though, all of this is a playground and I am most interested in the concepts, not the details.
You will also notice that I am only showing snippets of the multipath.conf file. The reason for this is simple: if you don’t follow your storage vendor’s documentation detailing how to configure dm-multipath for the storage system in use, you might run into issues. Always consult your storage vendor’s official documentation!
Installing dm-multipath
I didn’t have multipathing installed initially and had to install it first. Here is the transcript:
[root@server6 ~]# yum install device-mapper-multipath [...] --> Running transaction check ---> Package boost-system.x86_64 0:1.53.0-27.el7 will be installed ---> Package boost-thread.x86_64 0:1.53.0-27.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================================ Package Arch Version Repository Size ================================================================================================ Installing: device-mapper-multipath x86_64 0.4.9-111.el7_4.2 ol7_latest 134 k Installing for dependencies: boost-system x86_64 1.53.0-27.el7 ol7_latest 39 k boost-thread x86_64 1.53.0-27.el7 ol7_latest 57 k device-mapper-multipath-libs x86_64 0.4.9-111.el7_4.2 ol7_latest 251 k librados2 x86_64 1:0.94.5-2.el7 ol7_latest 1.7 M Updating for dependencies: kpartx x86_64 0.4.9-111.el7_4.2 ol7_latest 72 k Transaction Summary ================================================================================================ Install 1 Package (+4 Dependent packages) Upgrade ( 1 Dependent package) Total download size: 2.2 M [...] Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : boost-system-1.53.0-27.el7.x86_64 1/7 Installing : boost-thread-1.53.0-27.el7.x86_64 2/7 Installing : 1:librados2-0.94.5-2.el7.x86_64 3/7 Installing : device-mapper-multipath-libs-0.4.9-111.el7_4.2.x86_64 4/7 Updating : kpartx-0.4.9-111.el7_4.2.x86_64 5/7 Installing : device-mapper-multipath-0.4.9-111.el7_4.2.x86_64 6/7 Cleanup : kpartx-0.4.9-111.el7.x86_64 7/7 Verifying : boost-system-1.53.0-27.el7.x86_64 1/7 Verifying : 1:librados2-0.94.5-2.el7.x86_64 2/7 Verifying : boost-thread-1.53.0-27.el7.x86_64 3/7 Verifying : device-mapper-multipath-0.4.9-111.el7_4.2.x86_64 4/7 Verifying : kpartx-0.4.9-111.el7_4.2.x86_64 5/7 Verifying : device-mapper-multipath-libs-0.4.9-111.el7_4.2.x86_64 6/7 Verifying : kpartx-0.4.9-111.el7.x86_64 7/7 Installed: device-mapper-multipath.x86_64 0:0.4.9-111.el7_4.2 Dependency Installed: boost-system.x86_64 0:1.53.0-27.el7 boost-thread.x86_64 0:1.53.0-27.el7 device-mapper-multipath-libs.x86_64 0:0.4.9-111.el7_4.2 librados2.x86_64 1:0.94.5-2.el7 Dependency Updated: kpartx.x86_64 0:0.4.9-111.el7_4.2 Complete! [root@server6 ~]#
First configuration attempt
Next I quickly created a config file based on what I used in the past, making sure I add getuid_callout in the defaults {} section. And sure enough, when starting the service it complains about illegal syntax:
[root@server6 ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2018-04-03 14:56:19 BST; 1min 28s ago
Process: 586 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
Process: 568 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
Process: 564 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
Main PID: 596 (multipathd)
CGroup: /system.slice/multipathd.service
└─596 /sbin/multipathd
Apr 03 14:56:19 server6 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Apr 03 14:56:19 server6 multipath[568]: Apr 03 14:56:19 | /etc/multipath.conf line 24, invalid keyword: getuid_callout
Apr 03 14:56:19 server6 systemd[1]: Started Device-Mapper Multipath Device Controller.
Apr 03 14:56:19 server6 multipathd[596]: /etc/multipath.conf line 24, invalid keyword: getuid_callout
Note the line reading “/etc/multipath.conf line 24, invalid keyword: getuid_callout”. It didn’t seem to prevent the meta-device from being created, but I didn’t like the error message popping up all the time.
After removing the getuid_callout from my multipath.conf file to let the defaults take over, the error went away after a restart of the service:
Apr 03 15:02:54 server6 systemd[1]: Stopped Device-Mapper Multipath Device Controller. Apr 03 15:10:43 server6 systemd[1]: Starting Device-Mapper Multipath Device Controller... Apr 03 15:10:43 server6 systemd[1]: Started Device-Mapper Multipath Device Controller. Apr 03 15:10:43 server6 multipathd[11593]: mpatha: load table [0 4194304 multipath 0 0 1 1 service-time 0 1 1 8:0 1] Apr 03 15:10:43 server6 multipathd[11593]: mpatha: event checker started Apr 03 15:10:43 server6 multipathd[11593]: path checkers start up
So I guess I have to rethink my approach to dm-multipath and device name persistence if I want to avoid this error message.
The New Way of doing things
I have once been told by a good friend: “when unsure, consult the documentation”. On my Linux system, my first stop is the man page. After a quick “man multipath.conf” I learned that getuid_callout is nowhere to be found. The SUSE note mentioned a new attribute, named uid_attribute, which seems to have taken its place:
- uid_attribute:
- The udev attribute providing a unique path identifier. Default value is ID_SERIAL
I also found this to be valuable about the “multipaths” section:
This section defines the multipath topologies. They are indexed by a World Wide Identifier (wwid), which is taken to be the value of the udev attribute given by the uid_attribute keyword.
Aha! The new model is potentially much more flexible than the old one.
Controlling aliases with multipath {} sections
I really like to assign alias names to LUNs mapped to my database systems. This way, I know that a device named “/dev/mapper/oracle_asm_data_001” is most likely used as the first ASM disk in disk group DATA. If I see /dev/mapper/mpatha instead, I know precisely nothing about its purpose.
Giving LUNs a name is not hard, you can do this in the multipaths {} section of /etc/multipath.conf. I really like doing this for reasons just mentioned. Which options do I have?
The easiest one is quite simply a change of getuid_callout to uid_attribute or its removal. Thankfully – at least in my case – the output of scsi_id and the ID_SERIAL attribute provided by udev are nearly identical:
[root@server6 ~]# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda 0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 [root@server6 ~]# udevadm info /dev/sda | grep ID_SERIAL E: ID_SERIAL=0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 E: ID_SERIAL_SHORT=drive-scsi0-0-0-0 [root@server6 ~]#
Note that I included the –replace-whitespace option in the scsi_id command, otherwise the output wouldn’t be identical. This is a concession to my lab, on real hardware I haven’t seen white space in WWIDs (this doesn’t mean there aren’t). Removing the getuid_callout from the default {} section was actually enough to create a persistent device name.
[...]
multipaths {
multipath {
wwid "0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0"
alias "oracle_asm_data_001"
}
}
And yes, I get my alias after restarting the multipathing daemon.
Apr 03 16:40:49 server6 systemd[1]: Starting Device-Mapper Multipath Device Controller... Apr 03 16:40:49 server6 systemd[1]: Started Device-Mapper Multipath Device Controller. Apr 03 16:40:49 server6 multipathd[12271]: oracle_asm_data_001: load table [0 4194304 multipath 0 0 1 1 service-time 0 1 1 8:0 1] Apr 03 16:40:49 server6 multipathd[12271]: oracle_asm_data_001: event checker started Apr 03 16:40:49 server6 multipathd[12271]: path checkers start up [root@server6 ~]# multipath -l oracle_asm_data_001 (0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0) dm-3 QEMU ,QEMU HARDDISK size=2.0G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active `- 0:0:0:0 sda 8:0 active undef unknown [root@server6 ~]#
There is of course no second path, and there is a fair bit of information missing that your would otherwise get from an enterprise deployment, but you get the idea.
