This post is long overdue, and it’s as much a reminder to myself as it is a post for you to find when scratching your head.
Here’s the situation: Oracle Restart on an Oracle Enterprise Linux 5.5 system with ASMLib in use. I have installed 11.2.0.2 plus the latest PSU to bring database and Grid Infrastructure to 11.2.0.2.5-so far so good. I have used the non-GUI approach and everything seemed ok until I tried to create a database for some benchmarking. Here’s the surprising output:
[oracle@server1 create_database_kit]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.2.0 Production on Thu Apr 12 09:25:58 2012 Copyright (c) 1982, 2010, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production With the Partitioning, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> @cr_db ORACLE instance started. Total System Global Area 4275781632 bytes Fixed Size 2233336 bytes Variable Size 822086664 bytes Database Buffers 3422552064 bytes Redo Buffers 28909568 bytes create database SLOB CONTROLFILE REUSE * ERROR at line 1: ORA-03113: end-of-file on communication channel Process ID: 32745 Session ID: 199 Serial number: 1 ERROR: ORA-03114: not connected to ORACLE ERROR: ORA-03114: not connected to ORACLE Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options
Huh? Checking the HAS (high availability service) it was up. Also, all the resources showed green in crsctl. So what gives? As always, when checking database problems you should have a look at the corresponding alert.log entries. Here they come:
create database SLOB CONTROLFILE REUSE SET DEFAULT BIGFILE TABLESPACE maxinstances 1 maxdatafiles 1024 maxlogfiles 16 noarchivelog datafile size 50M logfile SIZE 4G, SIZE 4G, SIZE 4G user system identified by *user sys identified by WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command Default Temporary Tablespace will be necessary for a locally managed database in future release 2012-04-12 09:19:18.694000 +01:00 Starting background process ASMB ASMB started with pid=23, OS id=29916 Starting background process RBAL RBAL started with pid=24, OS id=29920 NOTE: initiating MARK startup Starting background process MARK MARK started with pid=25, OS id=29922 NOTE: MARK has subscribed 2012-04-12 09:19:22.014000 +01:00 Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_rbal_29920.trc: ORA-15183: ASMLIB initialization error [driver/agent not installed] WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_rbal_29920.trc: ORA-15183: ASMLIB initialization error [driver/agent not installed] SUCCESS: diskgroup DATA was mounted Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_ora_29910.trc (incident=177): ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/product/diag/rdbms/slob/SLOB/incident/incdir_177/SLOB_ora_29910_i177.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2012-04-12 09:19:24.138000 +01:00 Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_ora_29910.trc (incident=178): ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/product/diag/rdbms/slob/SLOB/incident/incdir_178/SLOB_ora_29910_i178.trc Dumping diagnostic data in directory=[cdmp_20120412091924], requested by (instance=1, osid=29910), summary=[incident=177]. 2012-04-12 09:19:25.128000 +01:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 29910 Dumping diagnostic data in directory=[cdmp_20120412091925], requested by (instance=1, osid=29910), summary=[incident=178]. 2012-04-12 09:20:21.060000 +01:00 Sweep [inc][178]: completed Sweep [inc][177]: completed Sweep [inc2][178]: completed Sweep [inc2][177]: completed 2012-04-12 09:20:26.484000 +01:00 SUCCESS: diskgroup DATA was dismounted
These two lines really stick out:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Can you already guess what it is?
The message that oracle cannot access the ASMLib drivers is strange since I could happily access all the disk groups from within ASM. Just to make sure I mounted and unmounted disk groups and that worked. The output from “crsctl status resource” also indicated everything was in order. So what?
I should state at this point that I am using the grid user for Grid Infrastructure.
And then it dawned (finally, after having lost 10 minutes)… when running dbca to generate the database creation scripts, it executes the a command called setasmgidwrap as one of the first steps. Let’s try it to see if that solves my problem:
[grid@server1 bin]$ cd $ORACLE_HOME/bin [grid@server1 bin]$ ./setasmgidwrap o=/u01/app/oracle/product/11.2.0.2/bin/oracle
And fair enough, the create database command now completed ok:
Voila! Just remember to run setasmgidwrap as shown and you no longer get these ORA-600 errors!
Martins Blog 
Pingback: Log Buffer #267, A Carnival of the Vanities for DBAs | The Pythian Blog
Martin, just wanted to add to this. I was getting the same error message ORA-15183 under 12c while trying to create an Active Data Guard physical standby. I had to run the command “setasmgidwrap o=/u01/app/oracle/product/12.1.0/db_1/bin/oracle” as the grid user before it would actually create the standby. Thanks for this post, as it gave me the pointer I needed to fix my problem! ;-)