This is a post that highlights the difference between operating systems, but also the fact that sometime it is hard to break out of a habit once you got used to it. My background is that of a Linux enthusiast, even though I equally like Solaris and AIX but I have a little less exposure to those.
Background
I have recently been asked to look at RAC on SPARC, which I gladly did. The system I was given had the usual software stack for RAC at this customer’s site. It comprised of:
- Solaris 10 Update 9 64bit on SPARC
- EMC Power Path 5.1
- EMC VMAX storage – 10x10G LUNs for a specific performance test
The Power Path configuration has already been in place when I got the machine, and I was allocated /dev/emcpower2[0-9] for my ASM testing. For the experienced Linux user who relies on device-mapper-multipath, the Power Path naming convention can be a bit confusing at first. For reference, the pseudo devices we are interested in for ASM are created under /dev/rdsk/-the “raw” device directory for “character” based access rather than the block device in /dev/dsk/. By default, the Power Path devices are called “emcpower”, followed by a number and a letter (in SPARC). An example would be /dev/rdsk/emcpower20c.
The number (20) is just a sequence number and doesn’t have any other meaning as far as I know. It also doesn’t seem to be consistent across the cluster nodes by default. The suffix-letter (“c”) does have a meaning though: it indicates the slice of the device. For example, the latter “a” indicates slice 0, xxxc is the whole disk, and xxxg is slice 6.
You can use the format command to have a look at the slices currently defined. Simply type “partition” and then “print” to print it.
Now all the disks I was given had a slice 6, from cylinder 274 to the end of the disk:
# format [disk selection not shown here] format> partition partition> print Current partition table (original): Total disk cylinders available: 10238 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 0 - 136 128.44MB (137/0/0) 263040 1 swap wu 137 - 273 128.44MB (137/0/0) 263040 2 backup wu 0 - 10237 9.37GB (10238/0/0) 19656960 3 unassigned wm 0 0 (0/0/0) 0 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 usr wm 274 - 10237 9.12GB (9964/0/0) 19130880 7 unassigned wm 0 0 (0/0/0) 0 partition> quit
If you’d like to see how the disks are connected to the array, of if there are any faults, use powermt display dev=all, or alternatively look at a specific device:
# powermt display dev=emcpower20 Pseudo name=emcpower20a Symmetrix ID=00x2x4x0x6x4 Logical device ID=0x0x state=alive; policy=SymmOpt; priority=0; queued-IOs=0; ============================================================================== --------------- Host --------------- - Stor - -- I/O Path -- -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3072 pci@400/pci@0/pci@c/fibre-channel@0/fp@0,0 c2t50000974C00A6118d17s0 FA 7eA active alive 0 0 3077 pci@500/pci@0/pci@9/fibre-channel@0/fp@0,0 c3t50000974C00A611Cd17s0 FA 8eA active alive 0 0
If you care to learn more about how your system administrator configured the paths (I have 2 HBAs as you can see), use “powermt display”. Also note the disks c2t5….s0 and c3t5…s0 are the native devices aggregated into emcpower20a.
I have changed the permissions on the pseudo devices to 664 and made them owned by oracle:oinstall before launching OUI. In the disk discovery screen I changed the disk discovery string to /dev/rdsk/emcpower2* but didn’t see a single disk in the OUI window. That was odd.
Troubleshooting disk discovery
This error lead me to look at /tmp/OraInstall<date>/installActions<date>.log which referenced a call to kfod, the disk discovery tool. The entry in the log file showed this content:
WARNING: SRVCTL not found as there is no Grid Infrastructure home on the box.
INFO: Executing [/tmp/OraInstall2011-07-25_11-33-50AM/ext/bin/kfod, nohdr=true, verbose=true, disks=all, status=true\
op=disks, asm_diskstring='/dev/rdsk/emcpower2*']
with environment variables {XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt, DISPLAY=localhost:12.0,\
PWD=/u01/app/oracle/stage/grid, LC_CTYPE=en_GB.ISO8859-1, _=./runInstaller, USER=oracle,\
NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, LC_COLLATE=en_GB.ISO8859-1, CLASSPATH=, HOME=/export/home/oracle,\
LC_NUMERIC=en_GB.ISO8859-1, SSH_CONNECTION=10.128.46.56 52883 10.129.48.214 22,\
LD_LIBRARY_PATH=/tmp/OraInstall2011-07-25_11-33-50AM/ext/bin/../lib,\
ORACLE_HOME=/tmp/OraInstall2011-07-25_11-33-50AM/ext/bin, SHELL=/usr/bin/bash, LOGNAME=oracle,\
SSH_TTY=/dev/pts/3, SHLVL=1, LC_MONETARY=en_GB.ISO8859-1, MANPATH=/usr/share/man:/usr/openwin/share/man:\
/db/pub/man:/db/pub/infra/samba/current/man:/db/pub/infra/rsync/current/man:/db/pub/infra/rcs/current/man:\
/db/pub/infra/tcpdump/current/man:/usr/dt/share/man:/usr/java1.2/man:/usr/apache/man:/usr/perl5/man:/usr/j2se/man:\
/usr/local/man:/opt/SUNWconn/ge/man:/opt/SUNWconn/man:/opt/SUNWsan/man:/opt/VRTS/man:/opt/DBpam/man:\
/opt/SUNWexplo/man:/opt/SUNWcstu/man:/opt/VRTSvlic/man, SSH_CLIENT=10.128.46.56 52883 22, \
MAIL=/var/mail//oracle, TZ=GB-Eire, LC_TIME=en_GB.ISO8859-1, JAVA_HOME=, LC_MESSAGES=C, PS1=[\u@\h \W]> , \
OLDPWD=/u01/app/oracle/stage, LC_ALL=, TERM=xterm, TMOUT=0, PATH=/usr/bin:/usr/ccs/bin:/usr/bin:/opt/EMCpower/bin:\
/etc/emc/bin:/etc:/zones/app/oracle/stage/grid/install}
INFO: Starting Output Reader Threads for process /tmp/OraInstall2011-07-25_11-33-50AM/ext/bin/kfod
INFO: Parsing KFOD-00311: Error scanning device /dev/rdsk/c3t50000974C00A611Cd16s7
[more errors skipped]
..
(Re-) Discovery of KFOD
So therein lies the problem! As a side effect I rediscovered the command line options used with kfod! Maybe I can use this for some experiments… All you need to do is to set environment variables for ORACLE_HOME and LD_LIBRARY_PATH as per the above output.
After a little fiddling around I remembered that unlike the tests with ZFS I did before I can’t pass the whole disk to ASM (the VTOC is in the first part of the disk!), but rather have to use a slice. So by changing the disk string to /dev/rdsk/emcpower2*g I got a successful report from kfod, executed as “oracle”:
$ /tmp/OraInstall2011-07-25_11-33-50AM/ext/bin/kfod verbose=true, disks=all status=true op=disks \ > asm_diskstring='/dev/rdsk/emcpower2*g' -------------------------------------------------------------------------------- Disk Size Header Path User Group ================================================================================ 1: 9341 Mb CANDIDATE /dev/rdsk/emcpower20g oracle oinstall 2: 9341 Mb CANDIDATE /dev/rdsk/emcpower21g oracle oinstall 3: 9341 Mb CANDIDATE /dev/rdsk/emcpower22g oracle oinstall 4: 9341 Mb CANDIDATE /dev/rdsk/emcpower23g oracle oinstall 5: 9341 Mb CANDIDATE /dev/rdsk/emcpower24g oracle oinstall 6: 9341 Mb CANDIDATE /dev/rdsk/emcpower25g oracle oinstall 7: 9341 Mb CANDIDATE /dev/rdsk/emcpower26g oracle oinstall 8: 9341 Mb CANDIDATE /dev/rdsk/emcpower27g oracle oinstall 9: 9341 Mb CANDIDATE /dev/rdsk/emcpower28g oracle oinstall 10: 9341 Mb CANDIDATE /dev/rdsk/emcpower29g oracle oinstall KFOD-00311: Error scanning device /dev/rdsk/emcpower2g ORA-15025: could not open disk "/dev/rdsk/emcpower2g" SVR4 Error: 13: Permission denied Additional information: 42 Additional information: 272256 Additional information: 12699536
I haven’t worked out which wildcard characters are valid in the ASM_DISKSTING (‘/dev/rdsk/emcpower2[0-9]g’ didn’t work), so I left it there. The emcpower2g device was not meant to be used in the test anyway.
With this setting I returned to the OUI window and continued the installation.
Lesson Learned
Oracle doesn’t always have useful logs, but in this case it was very useful to look at the install-actions file in /tmp. And also remember that creating ASM disks is different from Linux in Solaris and other platforms.
With Powerpath to display all your devices you can use the command
powermt display dev=all
Export your mappings between systems to ensure powerpath naming is consistent across nodes
emcpadm export_mappings -f /tmp/mappings.exp
On other nodes
emcpadm import_mappings -f /tmp/mappings.exp
There is an MyoS note ASM does not discover disk on Solaris platform 368840.1
– disks should be formatted from a slice(eg slice 0) other than S2 [Powerpath uses a-h for slices]
– partition should be of type usr and starting at cylinder 3 ( needs 512kB for VToC)
– disks should be labeled
– change ownership of the device file to grid:oinstall
– permissions on the device file should be 660
thank you Martin, I managed to find good clues with your post !