ASM normal redundancy and failure groups spanning SANs
Posted by Martin Bach on April 6, 2011
Julian Dyke has started an interesting thread on the Oak Table mailing list after the latest UKOUG RAC and HA SIG. Unfortunately I couldn’t attend that event, I wish I had, and I knew it would be great.
Anyway, the question revolved around an ASM disk group created with normal redundancy spanning two storage arrays. This should in theory protect against the failure of an array, although at a high price. All ASM disks exported from an array would be 1 failure group. Remember that disks in a failure group all fail if the supporting infrastructure (network, HBA, controller etc) fails. So what would happen with such a setup, if you followed these steps:
- Shutdown the array for failure group 2
- Stop the database
- Shutdown the second array – failure group 1
- Do some more maintenance…
- Startup failgroup B SAN
- Start the database
- Startup failgroup A SAN
ASM can tolerate the failure of one failgroup (capacity permitting), so the failure of failgroup 2 should not bring the disk group down, which would result in immediate loss of service. But what happens if it comes up after the data in the other failure group has been modified? Will there be data corruption?
Replaying
To simulate two storage arrays my distinguished filer01 and filer02 OpenFiler appliances have been used, each exporting 2 approx. 4G “LUNS” to my database host. At this time I only had access to my 11.1.0.7 2 node RAC system, if time permits I’ll repeat this with 10.2.0.5 and 11.2.0.2. The RAC cluster in the SIG presentation was 10.2. I am skipping the bit about the LUN creation and presentation to the hosts, and assume the following setup:
[root@rac11gr1node1 ~]# iscsiadm -m session tcp: [1] 192.168.99.51:3260,1 iqn.2006-01.com.openfiler:filer02DiskB tcp: [2] 192.168.99.50:3260,1 iqn.2006-01.com.openfiler:filer01DiskA tcp: [3] 192.168.99.51:3260,1 iqn.2006-01.com.openfiler:filer02DiskA tcp: [4] 192.168.99.50:3260,1 iqn.2006-01.com.openfiler:filer01DiskB
192.168.99.50 is my first openfiler instance, .192.168.99.51 the second. As you can see each export DISKA and DISKB. Mapped to the hosts, this is the target mapping (use iscsiadm –mode session –print 3 to find out):
- filer02DiskB: /dev/sda
- filer01DiskA: /dev/sdb
- filer02DiskA: /dev/sdc
- filer01DiskB: /dev/sdd
I am using ASMLib (as always on the lab) to label these disks:
[root@rac11gr1node1 ~]# oracleasm listdisks DATA1 DATA2 FILER01DISKA FILER01DISKB FILER02DISKA FILER02DISKB
DATA1 and DATA2 will not play a role in this article, I’m interested in the other disks. Assuming that the scandisks command completed on all nodes, I can add the disks to the new diskgroup:
SQL> select path from v$asm_disk PATH -------------------------------------------------------------------------------- ORCL:FILER01DISKA ORCL:FILER01DISKB ORCL:FILER02DISKA ORCL:FILER02DISKB ORCL:DATA1 ORCL:DATA2
Let’s create the diskgroup. The important part is to create failure groups per storage array. By the way this is not different from extended distance RAC!
SQL> create diskgroup fgtest normal redundancy 2 failgroup filer01 disk 'ORCL:FILER01DISKA', 'ORCL:FILER01DISKB' 3 failgroup filer02 disk 'ORCL:FILER02DISKA', 'ORCL:FILER02DISKB' 4 attribute 'compatible.asm'='11.1'; Diskgroup created.
With that done let’s have a look at the asm disk information:
SQL> select MOUNT_STATUS,HEADER_STATUS,STATE,REDUNDANCY,FAILGROUP,PATH from v$asm_disk where group_number=2; MOUNT_S HEADER_STATU STATE REDUNDA FAILGROUP PATH ------- ------------ -------- ------- ------------------------------ -------------------- CACHED MEMBER NORMAL UNKNOWN FILER01 ORCL:FILER01DISKA CACHED MEMBER NORMAL UNKNOWN FILER01 ORCL:FILER01DISKB CACHED MEMBER NORMAL UNKNOWN FILER02 ORCL:FILER02DISKA CACHED MEMBER NORMAL UNKNOWN FILER02 ORCL:FILER02DISKB
I have set the disk repair time to 24 hours and raised compatible parameters for RDBMS and ASM to 11.1, resulting in these attributes:
SQL> select * from v$asm_attribute NAME VALUE GROUP_NUMBER ATTRIBUTE_INDEX ATTRIBUTE_INCARNATION READ_ON SYSTEM_ ------------------------------ -------------------- ------------ --------------- --------------------- ------- ------- disk_repair_time 3.6h 2 0 1 N Y au_size 1048576 2 8 1 Y Y compatible.asm 11.1.0.0.0 2 20 1 N Y compatible.rdbms 11.1.0.0.0 2 21 1 N Y
Unlike 11.2 where disk groups are managed as Clusterware resource, 11.1 requires you to manually start them or append the new disk group to ASM_DISKGORUPS. You should query gv$asm_diskgroup.state to ensure the new diskgroup is mounted on all cluster nodes.
I need some data! A small demo database can be restored to the new failure group to provide some experimental playground. This is quite easily done by using an RMAN duplicate with the correct {db|log}_file_name_convert parameter set.
Mirror
The diskgroup is created with normal redundancy, which means that ASM will create a mirror for every primary extent, taking failure groups into consideration. I wanted to ensure that the data is actually mirrored on the new disk group, which has group number 2.I need to get this information from V$ASM_FILE and V$ASM_ALIAS:
SQL> select * from v$asm_file where group_number = 2 GROUP_NUMBER FILE_NUMBER COMPOUND_INDEX INCARNATION BLOCK_SIZE BLOCKS BYTES SPACE TYPE REDUND STRIPE CREATION_ MODIFICAT R ------------ ----------- -------------- ----------- ---------- ---------- ---------- ---------- -------------------- ------ ------ --------- --------- - 2 256 33554688 747669775 16384 1129 18497536 78643200 CONTROLFILE HIGH FINE 05-APR-11 05-APR-11 U 2 257 33554689 747669829 8192 69769 571547648 1148190720 DATAFILE MIRROR COARSE 05-APR-11 05-APR-11 U 2 258 33554690 747669829 8192 60161 492838912 990904320 DATAFILE MIRROR COARSE 05-APR-11 05-APR-11 U 2 259 33554691 747669829 8192 44801 367009792 739246080 DATAFILE MIRROR COARSE 05-APR-11 05-APR-11 U 2 260 33554692 747669831 8192 25601 209723392 424673280 DATAFILE MIRROR COARSE 05-APR-11 05-APR-11 U 2 261 33554693 747669831 8192 641 5251072 12582912 DATAFILE MIRROR COARSE 05-APR-11 05-APR-11 U 2 262 33554694 747670409 512 102401 52429312 120586240 ONLINELOG MIRROR FINE 05-APR-11 05-APR-11 U 2 263 33554695 747670409 512 102401 52429312 120586240 ONLINELOG MIRROR FINE 05-APR-11 05-APR-11 U 2 264 33554696 747670417 512 102401 52429312 120586240 ONLINELOG MIRROR FINE 05-APR-11 05-APR-11 U 2 265 33554697 747670417 512 102401 52429312 120586240 ONLINELOG MIRROR FINE 05-APR-11 05-APR-11 U 2 266 33554698 747670419 8192 2561 20979712 44040192 TEMPFILE MIRROR COARSE 05-APR-11 05-APR-11 U 11 rows selected. SQL> select * from v$asm_alias where group_NUMBER=2 NAME GROUP_NUMBER FILE_NUMBER FILE_INCARNATION ALIAS_INDEX ALIAS_INCARNATION PARENT_INDEX REFERENCE_INDEX A S ------------------------------ ------------ ----------- ---------------- ----------- ----------------- ------------ --------------- - - RAC11G 2 4294967295 4294967295 0 3 33554432 33554485 Y Y CONTROLFILE 2 4294967295 4294967295 53 3 33554485 33554538 Y Y current.256.747669775 2 256 747669775 106 3 33554538 50331647 N Y DATAFILE 2 4294967295 4294967295 54 1 33554485 33554591 Y Y SYSAUX.257.747669829 2 257 747669829 159 1 33554591 50331647 N Y SYSTEM.258.747669829 2 258 747669829 160 1 33554591 50331647 N Y UNDOTBS1.259.747669829 2 259 747669829 161 1 33554591 50331647 N Y UNDOTBS2.260.747669831 2 260 747669831 162 1 33554591 50331647 N Y USERS.261.747669831 2 261 747669831 163 1 33554591 50331647 N Y ONLINELOG 2 4294967295 4294967295 55 1 33554485 33554644 Y Y group_1.262.747670409 2 262 747670409 212 1 33554644 50331647 N Y group_2.263.747670409 2 263 747670409 213 1 33554644 50331647 N Y group_3.264.747670417 2 264 747670417 214 1 33554644 50331647 N Y group_4.265.747670417 2 265 747670417 215 1 33554644 50331647 N Y TEMPFILE 2 4294967295 4294967295 56 1 33554485 33554697 Y Y TEMP.266.747670419 2 266 747670419 265 1 33554697 50331647 N Y
My USERS tablespace which I am interested in most has file number 261-I chose it for this example as it’s only 5M in size. Taking my 1 MB allocation unit into account, it means I don’t have to trawl through thousands of line of output when getting the extent map.
Credit where credit is due-the next queries are partly based on the excellent work by Luca Canali from CERN, who has looked at ASM internals for a while. Make sure you have a look at the excellent reference available here: https://twiki.cern.ch/twiki/bin/view/PDBService/ASM_Internals. So to answer the question if the extents making up my users tablespace we need to have a look at the X$KFFXP, i.e. file extent pointers view:
SQL> select GROUP_KFFXP,DISK_KFFXP,AU_KFFXP from x$kffxp where number_kffxp=261 and group_kffxp=2 order by disk_kffxp; GROUP_KFFXP DISK_KFFXP AU_KFFXP ----------- ---------- ---------- 2 0 864 2 0 865 2 0 866 2 1 832 2 1 831 2 1 833 2 2 864 2 2 866 2 2 865 2 3 832 2 3 833 2 3 831 12 rows selected.
As you can see, I have a number of extents, all evenly spread over my disks. I can verify that this information is correct by querying the X$KFDAT view as well which contains similar information, but more related to the disk to AU mapping
SQL> select GROUP_KFDAT,NUMBER_KFDAT,AUNUM_KFDAT from x$kfdat where fnum_kfdat = 261 and group_kfdat=2 GROUP_KFDAT NUMBER_KFDAT AUNUM_KFDAT ----------- ------------ ----------- 2 0 864 2 0 865 2 0 866 2 1 831 2 1 832 2 1 833 2 2 864 2 2 865 2 2 866 2 3 831 2 3 832 2 3 833 12 rows selected.
OK so I am confident that my data is actually mirrored-otherwise the following test would not make any sense. I have double checked that the disks in failgroup FILER01 actually belong to my OpenFiler “filer01”, and the same for filer02. Going back to the original scenario:
Shut down Filer02
This will take down all the disks of failure group B. Two minutes after taking the filer down I checked if it was indeed shut down:
martin@dom0:~> sudo xm list | grep filer filer01 183 512 1 -b---- 1159.6 filer02 512 1 1179.6 filer03 185 512 1 -b---- 1044.4
Yes, no doubt about it-it’s down. What would the effect be? Surely I/O errors, but I wanted to enforce a check. Connected to +ASM2 I issued the “select * from v$asm_disk” command. This caused quite significant logging in the instance’s alert.log:
NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Apr 06 17:17:39 2011 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKA disk:0x0.0x97954459 au:0 iop:0x2b2997b61330 bufp:0x2b29977b3e00 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKB disk:0x1.0x9795445a au:0 iop:0x2b2997b61220 bufp:0x2b29977b0200 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 Wed Apr 06 17:17:58 2011 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKB disk:0x1.0x9795445a au:0 iop:0x2b2997b61440 bufp:0x2b29977a6400 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKA disk:0x0.0x97954459 au:0 iop:0x2b2997b61550 bufp:0x2b29977b1600 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 Wed Apr 06 17:18:03 2011 WARNING: Disk (FILER02DISKA) will be dropped in: (86400) secs on ASM inst: (2) WARNING: Disk (FILER02DISKB) will be dropped in: (86400) secs on ASM inst: (2) GMON SlaveB: Deferred DG Ops completed. Wed Apr 06 17:19:26 2011 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKB disk:0x1.0x9795445a au:0 iop:0x2b2997b61550 bufp:0x2b29977b1600 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKA disk:0x0.0x97954459 au:0 iop:0x2b2997b61440 bufp:0x2b29977b0200 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 Wed Apr 06 17:20:10 2011 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKA disk:0x0.0x97954459 au:0 iop:0x2b2997b61000 bufp:0x2b29977a6400 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:0, diskname:ORCL:FILER02DISKB disk:0x1.0x9795445a au:0 iop:0x2b2997b61550 bufp:0x2b29977b1600 offset(bytes):0 iosz:4096 operation:1(Read) synchronous:0 result: 4 osderr:0x3 osderr1:0x2e pid:6690 Wed Apr 06 17:21:07 2011 WARNING: Disk (FILER02DISKA) will be dropped in: (86217) secs on ASM inst: (2) WARNING: Disk (FILER02DISKB) will be dropped in: (86217) secs on ASM inst: (2) GMON SlaveB: Deferred DG Ops completed. Wed Apr 06 17:27:15 2011 WARNING: Disk (FILER02DISKA) will be dropped in: (85849) secs on ASM inst: (2) WARNING: Disk (FILER02DISKB) will be dropped in: (85849) secs on ASM inst: (2) GMON SlaveB: Deferred DG Ops completed.
The interesting lines are “all mirror sides found readable, no repair required”. So taking down the failgroup didn’t cause an outage. The other ASM instance complained as well a little later:
2011-04-06 17:16:58.393000 +01:00 NOTE: initiating PST update: grp = 2, dsk = 2, mode = 0x15 NOTE: initiating PST update: grp = 2, dsk = 3, mode = 0x15 kfdp_updateDsk(): 24 kfdp_updateDskBg(): 24 WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:2, diskname:ORCL:FILER02DISKA disk:0x2.0x97a6d9f7 au:1 iop:0x2b9ea4855e70 bufp:0x2b9ea4850a00 offset(bytes):1052672 iosz:4096 operation:2(Write) synchronous:1 result: 4 osderr:0x3 osderr1:0x2e pid:870 NOTE: group FGTEST: updated PST location: disk 0000 (PST copy 0) 2011-04-06 17:17:03.508000 +01:00 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds NOTE: PST update grp = 2 completed successfully NOTE: initiating PST update: grp = 2, dsk = 2, mode = 0x1 NOTE: initiating PST update: grp = 2, dsk = 3, mode = 0x1 kfdp_updateDsk(): 25 kfdp_updateDskBg(): 25 2011-04-06 17:17:07.454000 +01:00 NOTE: group FGTEST: updated PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 2 completed successfully NOTE: cache closing disk 2 of grp 2: FILER02DISKA NOTE: cache closing disk 3 of grp 2: FILER02DISKB SUCCESS: extent 0 of file 267 group 2 repaired by offlining the disk NOTE: repairing group 2 file 267 extent 0 SUCCESS: extent 0 of file 267 group 2 repaired - all mirror sides found readable, no repair required 2011-04-06 17:19:04.526000 +01:00 GMON SlaveB: Deferred DG Ops completed. 2011-04-06 17:22:07.487000 +01:00 GMON SlaveB: Deferred DG Ops completed.
No interruption of service though, which is good-the GV$ASM_CLIENT view reported all database instances still connected.
SQL> select * from gv$asm_client; INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME STATUS SOFTWARE_VERSIO COMPATIBLE_VERS ---------- ------------ ---------------------------------------------------------------- -------- ------------ --------------- --------------- 2 2 rac11g2 rac11g CONNECTED 11.1.0.7.0 11.1.0.0.0 1 2 rac11g1 rac11g CONNECTED 11.1.0.7.0 11.1.0.0.0
The result in the V$ASM_DISK view was as follows:
SQL> select name,state,header_status,path from v$asm_disk; NAME STATE HEADER_STATU PATH FAILGROUP ------------------------------ -------- ------------ -------------------------------------------------- ------------------------------ NORMAL UNKNOWN ORCL:FILER02DISKA NORMAL UNKNOWN ORCL:FILER02DISKB DATA1 NORMAL MEMBER ORCL:DATA1 DATA1 DATA2 NORMAL MEMBER ORCL:DATA2 DATA2 FILER01DISKA NORMAL MEMBER ORCL:FILER01DISKA FILER01 FILER01DISKB NORMAL MEMBER ORCL:FILER01DISKB FILER01 FILER02DISKA NORMAL UNKNOWN FILER02 FILER02DISKB NORMAL UNKNOWN FILER02 8 rows selected.
As I expected the disks for failgroup filer02 are gone, and so is the information about the failure group. My disk repair time should be high enough to protect me from having to rebuild the whole disk group. Now I’m really curious if my database can become corrupted-I’ll increase the SCN.
[oracle@rac11gr1node1 ~]$ . setsid.sh rac11g [oracle@rac11gr1node1 ~]$ sq SQL*Plus: Release 11.1.0.7.0 - Production on Wed Apr 6 17:24:18 2011 Copyright (c) 1982, 2008, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP, Data Mining and Real Application Testing options SQL> select current_scn from v$database; CURRENT_SCN ----------- 5999304 SQL> begin 2 for i in 1..5 loop 3 execute immediate 'alter system switch logfile'; 4 end loop; 5 end; 6 / PL/SQL procedure successfully completed. SQL> select current_scn from v$database; CURRENT_SCN ----------- 5999378 SQL>
Back to the test case.
Stop the Database
[oracle@rac11gr1node1 ~]$ srvctl stop database -d rac11g [oracle@rac11gr1node1 ~]$ srvctl status database -d rac11g Instance rac11g2 is not running on node rac11gr1node1 Instance rac11g1 is not running on node rac11gr1node2
Done-this part was simple. Next they stopped their first filer. To prevent bad things from happening I’ll shut down ASM on all nodes. I hope that doesn’t invalidate the test but I can’t see how ASM would not get a problem if the other failgroup went down as well.
Shut down Filer01 and start Filer02
Also quite simple. Shutting down this filer will allow me to follow the story. After filer01 was down I started filer02. I’m curious as to how ASM will react. I have deliberately NOT put disk group FGTEST into the ASM_DISKSTRING, I want to start it manually to get a better understanding of what happens.
After having started ASM on both nodes, I queried V$ASM_DISK and tried to mount the disk group:
SQL> select disk_number,name,state,header_status,path,failgroup from v$asm_disk; DISK_NUMBER NAME STATE HEADER_STATU PATH FAILGROUP ----------- ------------------------------ -------- ------------ -------------------------------------------------- ------------------------------ 0 NORMAL MEMBER ORCL:FILER02DISKA 1 NORMAL MEMBER ORCL:FILER02DISKB 2 NORMAL UNKNOWN ORCL:FILER01DISKA 3 NORMAL UNKNOWN ORCL:FILER01DISKB 0 DATA1 NORMAL MEMBER ORCL:DATA1 DATA1 1 DATA2 NORMAL MEMBER ORCL:DATA2 DATA2 6 rows selected.
Ooops, now they are both gone….
SQL> alter diskgroup fgtest mount; alter diskgroup fgtest mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "1" is missing ORA-15042: ASM disk "0" is missing ORA-15080: synchronous I/O operation to a disk failed ORA-15080: synchronous I/O operation to a disk failed
OK, I have a problem here. Both ASM instances report I/O errors with the FGTEST diskgroup, and I can’t mount it. That means I can’t mount the database either-in a way it proves I won’t have corruption. But neither will I have a database, what is worse?
Can I get around this problem?
I think I’ll have to start filer01 and see if that makes a difference. Hopefully I can recover my system with the information in failgroup filer01. Soon after filer01 came online I tried the query against v$asmdisk again and tried to mount it.
SQL> select disk_number,name,state,header_status,path,failgroup from v$asm_disk; DISK_NUMBER NAME STATE HEADER_STATU PATH FAILGROUP ----------- ------------------------------ -------- ------------ -------------------------------------------------- ------------------------------ 0 NORMAL MEMBER ORCL:FILER02DISKA 1 NORMAL MEMBER ORCL:FILER02DISKB 2 NORMAL MEMBER ORCL:FILER01DISKA 3 NORMAL MEMBER ORCL:FILER01DISKB 0 DATA1 NORMAL MEMBER ORCL:DATA1 DATA1 1 DATA2 NORMAL MEMBER ORCL:DATA2 DATA2 6 rows selected.
That worked!
Wed Apr 06 17:45:32 2011 SQL> alter diskgroup fgtest mount NOTE: cache registered group FGTEST number=2 incarn=0x72c150d7 NOTE: cache began mount (first) of group FGTEST number=2 incarn=0x72c150d7 NOTE: Assigning number (2,0) to disk (ORCL:FILER01DISKA) NOTE: Assigning number (2,1) to disk (ORCL:FILER01DISKB) NOTE: Assigning number (2,2) to disk (ORCL:FILER02DISKA) NOTE: Assigning number (2,3) to disk (ORCL:FILER02DISKB) Wed Apr 06 17:45:33 2011 NOTE: start heartbeating (grp 2) kfdp_query(): 12 kfdp_queryBg(): 12 NOTE: cache opening disk 0 of grp 2: FILER01DISKA label:FILER01DISKA NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache opening disk 1 of grp 2: FILER01DISKB label:FILER01DISKB NOTE: cache opening disk 2 of grp 2: FILER02DISKA label:FILER02DISKA NOTE: F1X0 found on disk 2 fcn 0.0 NOTE: cache opening disk 3 of grp 2: FILER02DISKB label:FILER02DISKB NOTE: cache mounting (first) group 2/0x72C150D7 (FGTEST) Wed Apr 06 17:45:33 2011 * allocate domain 2, invalid = TRUE kjbdomatt send to node 0 Wed Apr 06 17:45:33 2011 NOTE: attached to recovery domain 2 NOTE: cache recovered group 2 to fcn 0.7252 Wed Apr 06 17:45:33 2011 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.7252 ABA NOTE: seq=3 blk=337 NOTE: cache mounting group 2/0x72C150D7 (FGTEST) succeeded NOTE: cache ending mount (success) of group FGTEST number=2 incarn=0x72c150d7 Wed Apr 06 17:45:33 2011 kfdp_query(): 13 kfdp_queryBg(): 13 NOTE: Instance updated compatible.asm to 11.1.0.0.0 for grp 2 SUCCESS: diskgroup FGTEST was mounted SUCCESS: alter diskgroup fgtest mount
The V$ASM_DISK view is nicely updated and everything seems to be green:
SQL> select disk_number,name,state,header_status,path,failgroup from v$asm_disk; DISK_NUMBER NAME STATE HEADER_STATU PATH FAILGROUP ----------- ------------------------------ -------- ------------ -------------------------------------------------- ------------------------------ 0 DATA1 NORMAL MEMBER ORCL:DATA1 DATA1 1 DATA2 NORMAL MEMBER ORCL:DATA2 DATA2 0 FILER01DISKA NORMAL MEMBER ORCL:FILER01DISKA FILER01 1 FILER01DISKB NORMAL MEMBER ORCL:FILER01DISKB FILER01 2 FILER02DISKA NORMAL MEMBER ORCL:FILER02DISKA FILER02 3 FILER02DISKB NORMAL MEMBER ORCL:FILER02DISKB FILER02 6 rows selected.
Brilliant-will it have an effect on the database?
Starting the Database
Even though things looked ok, they weren’t! I didn’t expect this to happen:
[oracle@rac11gr1node1 ~]$ srvctl start database -d rac11g PRKP-1001 : Error starting instance rac11g2 on node rac11gr1node1 rac11gr1node1:ora.rac11g.rac11g2.inst: rac11gr1node1:ora.rac11g.rac11g2.inst:SQL*Plus: Release 11.1.0.7.0 - Production on Wed Apr 6 17:48:58 2011 rac11gr1node1:ora.rac11g.rac11g2.inst: rac11gr1node1:ora.rac11g.rac11g2.inst:Copyright (c) 1982, 2008, Oracle. All rights reserved. rac11gr1node1:ora.rac11g.rac11g2.inst: rac11gr1node1:ora.rac11g.rac11g2.inst:Enter user-name: Connected to an idle instance. rac11gr1node1:ora.rac11g.rac11g2.inst: rac11gr1node1:ora.rac11g.rac11g2.inst:SQL> ORACLE instance started. rac11gr1node1:ora.rac11g.rac11g2.inst: rac11gr1node1:ora.rac11g.rac11g2.inst:Total System Global Area 1720328192 bytes rac11gr1node1:ora.rac11g.rac11g2.inst:Fixed Size 2160392 bytes rac11gr1node1:ora.rac11g.rac11g2.inst:Variable Size 1291847928 bytes rac11gr1node1:ora.rac11g.rac11g2.inst:Database Buffers 419430400 bytes rac11gr1node1:ora.rac11g.rac11g2.inst:Redo Buffers 6889472 bytes rac11gr1node1:ora.rac11g.rac11g2.inst:ORA-00600: internal error code, arguments: [kccpb_sanity_check_2], [9572], rac11gr1node1:ora.rac11g.rac11g2.inst:[9533], [0x000000000], [], [], [], [], [], [], [], [] rac11gr1node1:ora.rac11g.rac11g2.inst: rac11gr1node1:ora.rac11g.rac11g2.inst: rac11gr1node1:ora.rac11g.rac11g2.inst:SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production rac11gr1node1:ora.rac11g.rac11g2.inst:With the Partitioning, Real Application Clusters, OLAP, Data Mining rac11gr1node1:ora.rac11g.rac11g2.inst:and Real Application Testing options rac11gr1node1:ora.rac11g.rac11g2.inst: CRS-0215: Could not start resource 'ora.rac11g.rac11g2.inst'. PRKP-1001 : Error starting instance rac11g1 on node rac11gr1node2 CRS-0215: Could not start resource 'ora.rac11g.rac11g1.inst'.
Oops. A quick search on Metalink revealed note Ora-00600: [Kccpb_sanity_check_2], [3621501],[3621462] On Startup (Doc ID 435436.1). The explanation for the ORA-600 is that “the seq# of the last read block is higher than the seq# of the control file header block.” Oracle Support explains it with a lost write, but here the situation is quite different. Interesting! I have to leave that for another blog post.
SBayliss said
Excellent post.
gary (@garytas) said
Hi Martin,
It’s an excellent post.
Have you tried to disable Filer02 while some transactions are still running in database?
My testing showed that ASM normal redundancy with two SAN arrays causing transactions to freeze (query on v$asm_disk hangs).
ASM took “few minutes” to do re-silvering (based on asm log), before it allows the next query/transactions to complete.
It’s really unexpected result considering that ASM should provide redundancy with multiple storage arrays.
Hope you can share your finding in this environment.
Thank you.
Martin Bach said
Hi Gary,
I have to rebuild my test/lab environment on new hardware, will update the post then. At the same time I need to go to 11.2.0.3.2
Stay tuned.
Rajeev said
Hi Martin,
Wonderful post.
Can you please let me know if you were able to repair the database (ORA-600) in the end (awaiting your blog post as promised in end).
Thanks,
Rajeev
Martin Bach said
Hi Rajeev,
I can’t recall that off the top of my head, sorry.