After all the necessary work has been completed on the iSCSI appliance we are now ready to present the storage to the RAC nodes; I am using RHEL 5 for these. RHEL 5 and clones require a specific RPM to manage iSCSI storage, which is part of the distribution. Note that the package has a slightly different name in RHEL 4,and the steps to discover storage are slightly different to the ones presented here.
UPDATE 221219: as you may have guessed, both Red Hat Enterprise Linux 4 and 5 are now ancient history and things have changed in newer releases. This article is therefore archived and should not be referred to.
Installing the iSCSI initiator
The initiator, as the name implies, is required to initiate an iSCSI connections. You need to install the initiator if not done so already:
# rpm -ihv iscsi-initiator-utils-*.rpm
The RPM will install the software and also creates the central configuration directory, /etc/iscsi/ with all necessary configuration files. The example here assumes passwordless authentication, you’ll have to change /etc/iscsi/iscsid.conf if passwords are required for CHAP authentication. Maybe I’ll cover the security aspects in a later post, right now I won’t.
The first thing after installing the iscsi daemons is to start them:
[root@rhel5 ~]# service iscsi start iscsid dead but pid file exists Turning off network shutdown. Starting iSCSI daemon: [ OK ] [ OK ] Setting up iSCSI targets: iscsiadm: No records found! [ OK ] [root@rhel5 ~]#
iSCSI administration on the client
The iscsiadm tool is used for discovery, logging in, and deletion of targets. Once a target has been discovered, the iscsi startup scripts will try to mount it automatically.
But first, the target needs to be discovered for which we’ll use the iscsiadm tool. It comes in 3 modes, discovery, node and session. Use iscsiadm in discovery mode to discover the targets presented by the Openfiler appliance:
# iscsiadm --mode discovery --type sendtargets --portal openfiler [root@rhel5 ~]# iscsiadm -m discovery -t sendtargets -p openfiler 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.asm_001 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.ocr_a 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.ocr_b 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.vote_a 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.vote_b 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.vote_c
Since we set up all ACL correctly previously, the command should succeed. Note that the portal parameter accepts the name of the openfiler appliance, which is openfiler in my case. You could have used an IP address instead. If there are any issues, the debug option can be used to get to the bottom of these:
[root@rhel5 ~]# iscsiadm -m discovery -d 4 -t sendtargets -p openfiler iscsiadm: ip openfiler, port 3260, tgpt -1 iscsiadm: Max file limits 1024 1024 iscsiadm: starting sendtargets discovery, address openfiler:3260, iscsiadm: connecting to 192.168.220.140:3260 iscsiadm: connected local port 47796 to 192.168.220.140:3260 iscsiadm: connected to discovery address 192.168.220.140 iscsiadm: login response status 0000 iscsiadm: Could not open /var/lib/iscsi/send_targets/openfiler,3260 err 2 iscsiadm: discovery process to openfiler:3260 exiting 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.asm_001 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.ocr_a 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.ocr_b 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.vote_a 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.vote_b 192.168.220.140:3260,1 iqn.2009-10.com.openfiler:rac_vg.vote_c
Note that the maximum debug level is 8, but then you are really swamped with information so use the good old 2>&1 | less to view all of that. If that still doesn’t help, check that the host has permissions to query the iSCSI targets and also ensure that the netmask for a single host is 255.255.255.255. There might be firewall settings (iptables -L) in the way.
Log in to the target
Once the target is discovered, you can either manually log in to the target or let the iscsi startup scripts do it for you. The syntax for doing this manually is as follows:
# iscsiadm --mode node --portal openfiler --login --targetname <iqn>
For the ocr_a target the syntax could be as follows:
[root@rhel5 ~]# iscsiadm -m node -T iqn.2009-10.com.openfiler:rac_vg.ocr_a -l Logging in to [iface: default, target: iqn.2009-10.com.openfiler:rac_vg.ocr_a, portal: 192.168.220.140,3260] Login to [iface: default, target: iqn.2009-10.com.openfiler:rac_vg.ocr_a, portal: 192.168.220.140,3260]: successful
This repeatedly crashes domUs in Oracle VM Server 2.1.5! Unfortunately I don’t have a support contract so I can’t figure out why that happens but it seems that the dom0 is exclusively allowed to assign device letters which is exactly what the domU does when it discovers the iSCSI target.
The new LUN will appear as a new scsi device to Linux:
Oct 20 18:57:17 rhel5 iscsid: iSCSI logger with pid=7059 started! Oct 20 18:57:18 rhel5 iscsid: transport class version 2.0-724. iscsid version 2.0-868 Oct 20 18:57:18 rhel5 iscsid: iSCSI daemon with pid=7060 started! Oct 20 19:04:51 rhel5 kernel: scsi3 : iSCSI Initiator over TCP/IP Oct 20 19:04:52 rhel5 kernel: Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0 Oct 20 19:04:52 rhel5 kernel: Type: Direct-Access ANSI SCSI revision: 04 Oct 20 19:04:52 rhel5 kernel: SCSI device sdc: 131072 512-byte hdwr sectors (67 MB) Oct 20 19:04:52 rhel5 kernel: sdc: Write Protect is off Oct 20 19:04:52 rhel5 kernel: SCSI device sdc: drive cache: none Oct 20 19:04:52 rhel5 kernel: SCSI device sdc: 131072 512-byte hdwr sectors (67 MB) Oct 20 19:04:52 rhel5 kernel: sdc: Write Protect is off Oct 20 19:04:52 rhel5 kernel: SCSI device sdc: drive cache: none Oct 20 19:04:52 rhel5 kernel: sdc: unknown partition table Oct 20 19:04:52 rhel5 kernel: sd 3:0:0:0: Attached scsi disk sdc Oct 20 19:04:52 rhel5 kernel: sd 3:0:0:0: Attached scsi generic sg2 type 0 Oct 20 19:04:52 rhel5 iscsid: received iferror -38
Please ignore the size of the LUN-I am writing this on my laptop on the train home and needed a working example without wasting too much storage I don’t have :)
From this stage on on it’s exactly the same as any other scsi device: you use fdisk to create a partition on it, but don’t create a file system: these LUNs are for RAC and don’t require a file system. Unless of course you’d like to put OCR and voting disk on OCFS2 which I don’t want to do, I prefer ASM for these.
A final note on the automatic mode: after the devices are discovered using iscsiadm as described above, simply restart iscsi services and you’ll be logged in to all devices saving you quite a bit of typing. If you need more control over the process, set node.startup to manual in /etc/iscsi/iscsid.conf
This part of the article showed how to initiate target discovery and mounting. The iscsi daemons will take care of mounting the devices on the next client reboot, however, there is no device name stability yet which is important up until 11.2 (where OCR and vote disk go into ASM). The final article of the series will add some bits and bops about the different parts of the IQN, shows you how to unpresent a LUN and finally how to guarantee device name stability.