Category Archives: RMAN

Duplicate from the standby instead from the primary in 12c

This post is related to 12c and an active database duplication for a standby I did in my lab environment. I’d say although I first encountered it on 12c there is a chance you run into a similar situation with earlier releases too.

I would normally use ASM for all my databases to make my life easier but this time I had to be mindful of the available memory on the laptop-which at 8 GB-is not plenty. So I went with file system setup instead. After the initial preparations I was ready to launch the one-liner on the standby database:

RMAN> duplicate target database for standby from active database;

This worked away happily for a few moments only to come to an abrupt halt with the below error message. I have started the duplication process on the standby.

...
executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 01-APR-14
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00012 name=/u01/oradata/CDB1/valid/sysaux01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 04/01/2014 20:34:42
ORA-17628: Oracle error 19505 returned by remote Oracle server
ORA-19505: failed to identify file ""
channel ORA_DISK_1 disabled, job failed on it will be run on another channel
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 04/01/2014 20:34:42
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script

RMAN-03009: failure of backup command on ORA_DISK_1 channel at 04/01/2014 20:34:42
ORA-17628: Oracle error 19505 returned by remote Oracle server
ORA-19505: failed to identify file ""

Huh? File 12 is part of a PDB named VALID. It was definitely available on the file system on the source, and I even had a backup on disk for it… I first thought it had to do with the PDBs all closed on the source datbase but that was not the case. After opening the PDBs read-write I still had the same error. Strange. I thought about debugging and tracing RMAN but then decided to first try the duplication from the primary. Which was a good idea, because the primary actually gave me a more meaningful error message:

Starting restore at 19.03.2014 10:17:08
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cdb1
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00004 to /u01/oradata/STDBY1/undotbs01.dbf
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:25
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cdb1
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00005 to /u01/oradata/STDBY1/pdbseed/system01.dbf
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 03/19/2014 10:17:26
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-19849: error while reading backup piece from service cdb1
ORA-19504: failed to create file "/u01/oradata/STDBY1/pdbseed/system01.dbf"
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 1

So I forgot to create the needed directories on the standby. For consistency I created a subdirectory per PDB on the primary, which I forgot on the standby host. You can see this here:

RMAN> report schema;

Report of database schema for database with db_unique_name CDB1

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    790      SYSTEM               ***     /u01/oradata/CDB1/system01.dbf
3    750      SYSAUX               ***     /u01/oradata/CDB1/sysaux01.dbf
4    90       UNDOTBS1             ***     /u01/oradata/CDB1/undotbs01.dbf
5    250      PDB$SEED:SYSTEM      ***     /u01/oradata/CDB1/pdbseed/system01.dbf
6    5        USERS                ***     /u01/oradata/CDB1/users01.dbf
7    590      PDB$SEED:SYSAUX      ***     /u01/oradata/CDB1/pdbseed/sysaux01.dbf
11   270      VALID:SYSTEM         ***     /u01/oradata/CDB1/valid/system01.dbf
12   610      VALID:SYSAUX         ***     /u01/oradata/CDB1/valid/sysaux01.dbf
13   260      RCAT:SYSTEM          ***     /u01/oradata/CDB1/rcat/system01.dbf
14   610      RCAT:SYSAUX          ***     /u01/oradata/CDB1/rcat/sysaux01.dbf
15   100      RCAT:RMAN_CATALOG    ***     /u01/oradata/CDB1/rcat/rman_catalog01.dbf
16   50       VALID:USERS          ***     /u01/oradata/CDB1/valid/users01.dbf
17   250      DGTEST:SYSTEM        ***     /u01/oradata/CDB1/dgtest/system01.dbf
18   590      DGTEST:SYSAUX        ***     /u01/oradata/CDB1/dgtest/sysaux01.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    60       TEMP                 32767       /u01/oradata/CDB1/temp01.dbf
2    20       PDB$SEED:TEMP        32767       /u01/oradata/CDB1/pdbseed/pdbseed_temp01.dbf
4    20       VALID:TEMP           32767       /u01/oradata/CDB1/valid/pdbseed_temp01.dbf
5    20       RCAT:TEMP            32767       /u01/oradata/CDB1/rcat/pdbseed_temp01.dbf
6    20       DGTEST:TEMP          32767       /u01/oradata/CDB1/dgtest/dgtest_temp01.dbf

All right, after creating the directories I retried the duplication and it worked:

...
datafile 4 switched to datafile copy
input datafile copy RECID=5 STAMP=843771196 file name=/u01/oradata/STDBY1/undotbs01.dbf

datafile 5 switched to datafile copy
input datafile copy RECID=6 STAMP=843771198 file name=/u01/oradata/STDBY1/pdbseed/system01.dbf
datafile 6 switched to datafile copy
input datafile copy RECID=7 STAMP=843771198 file name=/u01/oradata/STDBY1/users01.dbf
datafile 7 switched to datafile copy
input datafile copy RECID=8 STAMP=843771199 file name=/u01/oradata/STDBY1/pdbseed/sysaux01.dbf
datafile 11 switched to datafile copy
input datafile copy RECID=9 STAMP=843771199 file name=/u01/oradata/STDBY1/valid/system01.dbf
datafile 12 switched to datafile copy
input datafile copy RECID=10 STAMP=843771199 file name=/u01/oradata/STDBY1/valid/sysaux01.dbf
datafile 13 switched to datafile copy
input datafile copy RECID=11 STAMP=843771199 file name=/u01/oradata/STDBY1/rcat/system01.dbf
datafile 14 switched to datafile copy
input datafile copy RECID=12 STAMP=843771200 file name=/u01/oradata/STDBY1/rcat/sysaux01.dbf
datafile 15 switched to datafile copy
input datafile copy RECID=13 STAMP=843771200 file name=/u01/oradata/STDBY1/rcat/rman_catalog01.dbf
datafile 16 switched to datafile copy
input datafile copy RECID=14 STAMP=843771200 file name=/u01/oradata/STDBY1/valid/users01.dbf
datafile 17 switched to datafile copy
input datafile copy RECID=15 STAMP=843771200 file name=/u01/oradata/STDBY1/dgtest/system01.dbf
datafile 18 switched to datafile copy
input datafile copy RECID=16 STAMP=843771200 file name=/u01/oradata/STDBY1/dgtest/sysaux01.dbf
Finished Duplicate Db at 19.03.2014 10:28:42

Summary

Although not a big problem and probably known by everyone besides me the error message on the standby shouldn’t be of the scary type ‘can’t identify datafile “”‘ followed by the process aborting. Instead something more meaningful would have been nice.

Recovering a standby over the network in 12c

Another one of the cool but underrated features in 12c is the possibility to recover a physical standby over the network with one line in RMAN.

Why do you need to perform this activity? Assume someone really clever created a segment “nologging” and the database was not in force logging mode. This operation cannot be replicated by redo apply on the standby, and you are bound to have a problem. Or, in my case, I had the standby shut down in my lab environment (intentionally) and created a few PDBs on my primary. For some reason I lost an archived redo log. This would of course not happen in a production environment, but my lab VM is limited when it comes to space and I may have moved my backup to a USB disk that I didn’t bring along.

So a long story short: you either recreate the standby-which is a rather blunt-force approach to the problem-or recover the standby from the SCN where the problem originated. I was hoping the new 12c recover database from service command in RMAN could help. It did, but not as I expected at first.

The problem with my lab

My environment is Oracle Linux 6.4 x86-64 on Virtual Box with 4 GB RAM. Oracle is 12.1.0.1.2. CDB1 is primary, CDB2 is my standby database. Both are on the same VM (I only have 8 GB total memory). CDB2 complained about a missing archived redo log:

SQL> select * from v$archive_gap;

   THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#     CON_ID
---------- ------------- -------------- ----------
         1            27             27          1

Continue reading

RMAN restore spfile from autobackup and compatible weirdness

Prompted by a recent interview I wanted to perform a little test with RMAN, and incrementally updated backups. I created a 11.2.0.2.5 database on my Linux (OEL 5.5) test system and refreshed my understanding of this most useful rman feature (which is for another post).

So after I was happy with the working of the incrementally updated image copies I decided to see if I could restore my database with those. Time to use “drop database”, which removes spfile, data files, temp files and the control files. Tabula rasa!

But it didn’t matter I thought: I have controlfile/spfile autobackups and fully recovered image copies plus all the archived logs in the FRA. What could possibly go wrong? Well it took me 30 minutes to get the database back. Continue reading

RMAN duplicate and read-only tablespaces

This one is to be filed under the “little things I didn’t really know until now” section: RMAN duplicate. Be honest to yourself: would you have known the answer to this question: what happens to tablespaces which are read-only on the source during an RMAN duplication (not for standby)?

Some background

I have started my career as a DBA on Oracle 8.1.7.0, and at the time not everyone fully embraced RMAN. OK, RMAN made it really hard at the time to fall in love with it. So when we “cloned” our production database to development, the following steps were followed:

  1. For each tablespace in the database, put it in hot backup mode and copy the files across to the destination location, then end the hot backup mode.
  2. Copy any archived logs generated in during the hot backup mode to DEV.
  3. Create a backup controlfile to trace.
  4. Edit the create controlfile statement (Set #2-resetlogs) and extract the “create controlfile” statement, adding the new database name in the “set” clause.
  5. Start DEV in nomount mode.
  6. Run create controlfile statement.
  7. Recover the database until cancel using the backup controlfile, applying all archived logs.
  8. Open the database with the resetlogs clause.

That worked ok, unless you had a data file offline or read only.

Continue reading

Recreate the standby controlfile when using ASM

This is in reply to some work I was supposed to carry out over the weekend on a DR database. It could have been really simple, but it turned out it wasn’t. As part of a major migration I moved a 2TB database from OCFS to ASM, which worked really nicely with a Data Guard switchover operation. Trouble now was that the standby databases still references online redo log files on the file system rather than ASM. I wanted to get rid of these annoying messages by updating the control file which was already rectified on the primary.

Under normal (non ASM) circumstances that doesn’t cause any trouble, but since ASM has the naming convention “+DISGROUPNAME/db_unique_name” the standby controlfile isn’t that easy to replace. Well actually it is but I needed some more careful thinking.

The steps are quite simple:

  • Create a standby controlfile on the primary: alter database create standby controlfile as ‘/tmp/primary.ctl’;
  • Ship this file to the standby database server
  • Create a backup of the existing standby controlfile just in case (rman: backup current controlfile format ‘/tmp/backup.ctl’)
  • Cancel managed recovery: alter database recover managed standby database cancel;
  • Shut the standby database down: shutdown immediate (if you’re a good citizen you also defer log shipping from the primary to avoid any errors poluting the alert.log and potentially triggering an alarm)
  • Startup nomount
  • RMAN: restore controlfile from ‘/tmp/primary.ctl’

A remark at this place: the controlfiles parameter usually uses OMF syntax when in ASM. Prior to 10.2.0.4 you couldn’t restore the controlfile if the spfile parameter was an OMF name, but it seems you can do this now. Users of other versions please reset the control_files parameter in the spfile and bounce the instance.

Continue reading

ORA-02290: check constraint (RMAN.AL_C_STATUS) violated

I ran into this problem today when I was working on a backup problem. The scenario is as follows:

  • Oracle 10.2.0.4 EE
  • RHEL 5.2 PAE kernel, i.e. 32bit
  • Backup taken on the standby database
  • Using a recovery catalog

All rman operations on the standby database with catalog database connection failed with the following error stack:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of list command at 10/30/2009 10:01:57
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of partial resync command on default channel at 10/30/2009 10:01:57
ORA-02290: check constraint (RMAN.AL_C_STATUS) violated

Ooops-so no backups, need to fix this quickly now. BTW, the same problem didn’t exist when running the same operation on the primary, but I don’t have space for the (disk) backup over there. Continue reading

Archivelog retention policy changes in RMAN 11g

In the past, that means at least until 10.2.x it used to be a bit clumsy to set the archivelog deletion policy to “applied on standby” on both primary and standby database. It involved setting an underscore parameter, as per Metalink Note 728053.1 “Configure RMAN to purge archivelogs after applied on standby”. For reference, the parameter was _log_deletion_policy, and you’d have to bounce the database for it to take effect.

SQL> alter system set "_log_deletion_policy"='ALL' scope=spfile;

Only then could you avoid the error message.Now with 11g that’s a lot easier.

Continue reading