WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

This post is long overdue, and it’s as much a reminder to myself as it is a post for you to find when scratching your head.

Here’s the situation: Oracle Restart on an Oracle Enterprise Linux 5.5 system with ASMLib in use. I have installed 11.2.0.2 plus the latest PSU to bring database and Grid Infrastructure to 11.2.0.2.5-so far so good. I have used the non-GUI approach and everything seemed ok until I tried to create a database for some benchmarking. Here’s the surprising output:

[oracle@server1 create_database_kit]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Apr 12 09:25:58 2012

Copyright (c) 1982, 2010, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> @cr_db
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size 2233336 bytes
Variable Size 822086664 bytes
Database Buffers 3422552064 bytes
Redo Buffers 28909568 bytes
create database SLOB CONTROLFILE REUSE
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 32745
Session ID: 199 Serial number: 1

ERROR:
ORA-03114: not connected to ORACLE

ERROR:
ORA-03114: not connected to ORACLE

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Huh? Checking the HAS (high availability service) it was up. Also, all the resources showed green in crsctl. So what gives? As always, when checking database problems you should have a look at the corresponding alert.log entries. Here they come:

create database SLOB CONTROLFILE REUSE
SET DEFAULT BIGFILE TABLESPACE
maxinstances 1
maxdatafiles 1024
maxlogfiles 16
noarchivelog
datafile size 50M
logfile SIZE 4G, SIZE 4G, SIZE 4G
user system identified by *user sys identified by
WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command
Default Temporary Tablespace will be necessary for a locally managed database in future release
2012-04-12 09:19:18.694000 +01:00
Starting background process ASMB
ASMB started with pid=23, OS id=29916
Starting background process RBAL
RBAL started with pid=24, OS id=29920
NOTE: initiating MARK startup
Starting background process MARK
MARK started with pid=25, OS id=29922
NOTE: MARK has subscribed
2012-04-12 09:19:22.014000 +01:00
Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_rbal_29920.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_rbal_29920.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup DATA was mounted
Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_ora_29910.trc (incident=177):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/product/diag/rdbms/slob/SLOB/incident/incdir_177/SLOB_ora_29910_i177.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2012-04-12 09:19:24.138000 +01:00
Errors in file /u01/app/oracle/product/diag/rdbms/slob/SLOB/trace/SLOB_ora_29910.trc (incident=178):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/product/diag/rdbms/slob/SLOB/incident/incdir_178/SLOB_ora_29910_i178.trc
Dumping diagnostic data in directory=[cdmp_20120412091924], requested by (instance=1, osid=29910), summary=[incident=177].
2012-04-12 09:19:25.128000 +01:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 29910
Dumping diagnostic data in directory=[cdmp_20120412091925], requested by (instance=1, osid=29910), summary=[incident=178].
2012-04-12 09:20:21.060000 +01:00
Sweep [inc][178]: completed
Sweep [inc][177]: completed
Sweep [inc2][178]: completed
Sweep [inc2][177]: completed
2012-04-12 09:20:26.484000 +01:00
SUCCESS: diskgroup DATA was dismounted

These two lines really stick out:

ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Can you already guess what it is?

The message that oracle cannot access the ASMLib drivers is strange since I could happily access all the disk groups from within ASM. Just to make sure I mounted and unmounted disk groups and that worked. The output from “crsctl status resource” also indicated everything was in order. So what?

I should state at this point that I am using the grid user for Grid Infrastructure.

And then it dawned (finally, after having lost 10 minutes)… when running dbca to generate the database creation scripts, it executes the a command called setasmgidwrap as one of the first steps. Let’s try it to see if that solves my problem:

[grid@server1 bin]$ cd $ORACLE_HOME/bin
[grid@server1 bin]$ ./setasmgidwrap o=/u01/app/oracle/product/11.2.0.2/bin/oracle

And fair enough, the create database command now completed ok:

Voila! Just remember to run setasmgidwrap as shown and you no longer get these ORA-600 errors!