Monthly Archives: April 2010

ORA-604 when starting the database

Interesting scenario this morning with a development database. There is no specific monitoring in place for development systems so a user phoned us up stating that the database was inaccessible. The last lines in the alert.log showed a problem with a datafile not readable:

Wed Apr 28 10:18:08 2010
Errors in file /u01/app/oracle/admin/devone/bdump/devone_pmon_3416.trc:
ORA-00376: file 2 cannot be read at this time
ORA-01110: data file 2: '+DATB/DEVONE/datafile/undotbs1.356.703506859'

This is a 10.2.0.4.1 single instance database (a clone from our production RAC cluster) running on RHEL 5.4 64bit with ASMLib. Continue reading

Advertisements

Huge Pages and Linux-Real World example

A common situation for many DBAs: all over sudden you are tasked to look at a database, and you are told you inherit it. Of course, a lot of problems exist with it, and you are supposed to fix them all. A few weeks ago this happened to me.

The system is a Sun 4660 x86-64 server with Red Hat5.3, and it has 64GB of memory, 8 dual core Opteron 8218 processors. SGA_TARGET was set to 40G, and PGA_AGGREGATE_TARGET to 5GB. That sounds like plenty, however the box was very busy trying to free up memory:

top - 12:16:11 up 23 days, 23:46, 19 users,  load average: 28.84, 25.69, 23.19
Tasks: 970 total,   3 running, 967 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.3%us, 13.0%sy,  0.0%ni, 36.0%id, 40.6%wa,  0.1%hi,  1.0%si,  0.0%st
Mem:  66068664k total, 65992772k used,    75892k free,    43168k buffers
Swap:  2096472k total,  2096472k used,        0k free, 40782300k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1068 root      20  -5     0    0    0 R 76.0  0.0 566:03.07 [kswapd5]
13284 oracle    18   0 40.3g 6.3g 6.2g D 31.5 10.0  28:30.12 oraclePROD (LOCAL=NO)
 1070 root      10  -5     0    0    0 D 21.6  0.0 925:57.03 [kswapd7]
 1065 root      10  -5     0    0    0 D 13.8  0.0 196:27.93 [kswapd2]
 7771 oracle    18   0 40.6g 4.5g 4.1g D 12.1  7.1  26:05.04 oraclePROD (LOCAL=NO)
 8073 oracle    16   0 40.2g 2.8g 2.8g D 12.1  4.4   5:29.79 oraclePROD (LOCAL=NO)
 1066 root      10  -5     0    0    0 S 11.8  0.0  84:57.85 [kswapd3]
 1067 root      10  -5     0    0    0 D 10.5  0.0 165:39.98 [kswapd4]
 1069 root      10  -5     0    0    0 S  7.9  0.0 277:15.84 [kswapd6]

Continue reading