One of the many interesting things I heard at the conference this time around was that Oracle’s future direction includes the use of database files on ACFS. When ACFS came out this was strictly ruled out, but has been possible for a little while now, I believe with 12.1.0.1.0. With the Oracle Database Appliance (ODA) using this deployment option and hearing about it at the conference, a little further investigation was in order. During one of the presentation @OracleRACPM Markus Michalewicz had a reference to a script that I didn’t know on his slides. The script is called gDBClone, and I wanted to see how it works. The idea is that the script can be used to create a snap-clone of a database if the source is on ACFS and in archivelog mode.
As it turned out there were a few hurdles along the way and I will point them out so you don’t run into the same issues.
UPDATE: these hurdles might be there since Oracle databases aren’t supported on ACFS in Oracle Restart: https://docs.oracle.com/database/121/OSTMG/asmfilesystem.htm#OSTMG95961 Please consider yourself warned! For the rest of the article let’s pretend that this is a clustered environment. The article is therefore purely educational and no encouragement to do this in real life.
The script and associated white paper assume that you have two systems-production and dev/test/uat. The goal of the clone procedure is to be able to create a copy of your live database on your dev/test/uat cluster. This database can then be used as the source for ACFS snapshots. This can be represented in ASCII art:
+------+ clone +------------------+
| PROD | ----> | MASTER |
+------+ | / \ | snap #1
| / \ | snap #2
| CLONE1 CLONE2 |
+------------------+
It is by no means required to follow this approach, and if nothing else then you can use the clone script to run a RMAN duplication in a single command. I once wrote a script to do the same but this was a truly complex thing to do.
For this article I’ll assume that you clone PROD (named ORCL) to MASTER, and snap MASTER to CLONE1.
The setup
Since my “travel-lab” is not a RAC cluster I opted for an installation of 12.1.0.2.1 for Oracle Restart and the database to keep it simple.
ACFS storage will be provided by and ADVM volume from disk group data. In my case this was quite easy to accomplish. Since this was an 12.1 system anyway I created my DATA disk group with ASM, RDBMS and ADVM compatibility for 12.1.0.2.0.
You create the ASM Dynamic Volume Manager (ADVM) volume on top of the ASM disk group, in my case on DATA. This is a simple task and can be performed by a variety of tools, I opted for a call to asmcmd:
ASMCMD> volcreate
usage: volcreate -G <diskgroup> -s <size> [ --column <number> ] [ --width <stripe_width> ]
[--redundancy {high|mirror|unprotected} ] [--primary {hot|cold}] [--secondary {hot|cold}] <volume>
help: help volcreate
ASMCMD> volcreate -G data -s 10G volume1
ASMCMD> volinfo --all
Diskgroup Name: DATA
Volume Name: VOLUME1
Volume Device: /dev/asm/volume1-162
State: ENABLED
Size (MB): 10240
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage:
Mountpath:
Initially I got a message that ASM could not communicate with the (ASM) Volume driver.
SQL> /* ASMCMD */alter diskgroup data add volume 'volume1' size 10G ORA-15032: not all alterations performed ORA-15477: cannot communicate with the volume driver ERROR: /* ASMCMD */alter diskgroup data add volume 'volume1' size 10G
This was solved by modprobe calls to the oracle kernel modules.
[root@server5 ~]# modprobe oracleacfs [root@server5 ~]# modprobe oracleadvm [root@server5 ~]# modprobe oracleoks [root@server5 ~]# lsmod | grep oracle oracleadvm 507006 7 oracleacfs 3307457 1 oracleoks 505749 2 oracleadvm,oracleacfs
There is a known issue with the execution of udev rules (/etc/udev/rules.d/55-usm.rules) that might delay the setting of permissions. On my system a udevadm trigger solved it. Still odd (Oracle Linux 6.6/ UEK 3 3.8.13-44.1.1.el6uek.x86_64), especially since a call to acfsdriverstate supported stated it was supported.
Once the volume is created it needs to be formatted using ACFS. This can be done in this way:
[oracle@server5 ~]$ mkfs -t acfs /dev/asm/volume1-162 mkfs.acfs: version = 12.1.0.2.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/volume1-162 mkfs.acfs: volume size = 10737418240 ( 10.00 GB ) mkfs.acfs: Format complete.
Trying to register the file system in the ACFS registry pointed me to the first problem with the procedure on Oracle Restart:
[root@server5 ~]# acfsutil registry -a /dev/asm/volume1-162 /u01/oradata
Usage: srvctl <command> <object> [<options>]
commands: enable|disable|start|stop|status|add|remove|modify|update|getenv|setenv|
unsetenv|config|upgrade|downgrade
objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
srvctl <command> -help [-compatible] or
srvctl <command> <object> -help [-compatible]
PRKO-2012 : filesystem object is not supported in Oracle Restart
acfsutil registry: ACFS-03111: unable to add ACFS mount /u01/oradata within Oracle Registry
Interesting-but not a problem in the lab. I usually mount file systems where in my opinion they logically belong to. In this case I mounted the file system to /u01/oradata. Spoiler alert: this is not what you are supposed to do if you want to use the gDBClone script.
To cut a long story short, the mount point was assumed to be in /acfs for a snap’d or cloned database. The script also assumes that your system is a RAC environment, and I found it not to work well in 12.1 at all due to the way it tries to get the database version from the OCR (OLR) profile. First the new ACFS file system is mounted, then made accessible to the oracle user:
[root@server5 ~]# mount -t acfs /dev/asm/volume1-162 /acfs [root@server5 ~]# chown -R oracle:dba /acfs [root@server5 ~]# mount | grep acfs /dev/asm/volume1-162 on /acfs type acfs (rw)
Creating the MASTER database
My source system resides in ASM, and there is no way of creating COW clones in ASM. The MASTER database must be moved to ACFS first as a result, from where you can take storage snapshots.
A quick hack was required since I only had 1 machine, so I created the source database (PROD in the ASCII art example) as “orcl” using the following call to dbca and setting it to archivelog mode:
dbca -silent -createDatabase -templateName General_Purpose.dbc -gdbname orcl \ > -sid orcl -sysPassword pwd1 -systemPassword pwd2 -emConfiguration none \ > -storageType ASM -asmsnmpPassword pwd3 -diskGroupName data -recoveryGroupName reco \ > -totalMemory 2048
The next step is to create the MASTER database. The gDBClone script checks if a database is on ACFS in function checkIfACFS(). If a database is found to be on ASM (by checking the SPFILE location in the Clusterware profile) it requires it to be CLONED as opposed to SNAPped. Here is the command to clone ORCL to MASTER. You must set your environment to an RDBMS home before executing the script.
# ./gDBClone clone -sdbname orcl -sdbhost server5 -sdbport 1521 -tdbname master -acfs /acfs -debug
It uses a RMAN duplicate under the covers. The arguments are almost self-explanatory. It takes the location of the source database (can be remote) and where you want to store the database. Since I desperately want to store the clone on ACFS I specified it in the command. The -debug flag prints more verbose output, a lot of information is also found in /var/log/gDBName/. Note that the script is to be run as root :(
To get there a few tricks were necessary in Oracle Restart environments, I’ll feed them back to Oracle to see if they can be added to the script. You probably won’t encounter problems when using a clustered 11.2 installation.
At one point the script checks the VERSION flag in the database resource profile (crsctl stat res ora..db -p | grep ‘^VERSION’, and since that field no longer shows up in 12.1 the variable is undefined in perl and the execution fails. The problem with Oracle Restart is related to setting the database type to SINGLE on the local host (srvctl add database … -c SINGLE -x …). Oracle Restart doesn’t understand those switches. Also you can’t set the db_unique_name in 12c to a name of a database already registered in the OCR. Moving the step to register the database further down in the execution helped.
Please don’t get me wrong: the script is very neat in that it allows you to run an RMAN duplicate command over the network, potentially creating the backup on the fly. In 12c RMAN will pull backup pieces if they exist instead of creating a backup on the fly to reduce its impact on the production database. Oh and I forgot-you can even convert it to a clustered database if it is not already.
Creating an ACFS clone database
Next up is the creation of the CLONE database. My first attempts were unsuccessful. The main trick seems to be to keep the ACFS mount underneath the / (root) file system. Mounting it elsewhere caused the script to fail. If you can read perl, check the $acfs variable and checkIfACFS() routine to understand why. Also, the compatibility of your ASM diskgroup containing the volume has to be greater than 11.2.0.3 or you get this error:
[root@server5 ~]# ./gDBClone snap -sdbname master -tdbname clone1
2014-12-12 11:03:53: I Getting host info...
2014-12-12 11:03:53: I Starting.....
2014-12-12 11:03:53: I Validating environment.....
Enter the source MASTER SYSDBA password:
2014-12-12 11:03:56: I Getting OH version...
2014-12-12 11:04:04: I Checking SCAN listener server5.example.com:1521...
2014-12-12 11:04:05: I Checking database CLONE1 existence...
2014-12-12 11:04:05: I Checking registered instance CLONE1 ...
2014-12-12 11:04:10: I Checking if the Source Database MASTER it's on ASM
2014-12-12 11:04:10: I Source Database MASTER it's on ACFS
2014-12-12 11:04:10: I Checking snapshot CLONE1 existence
2014-12-12 11:04:10: I Setting up clone environment....
2014-12-12 11:04:10: I Starting auxiliary listener....
2014-12-12 11:05:10: I Creating ACFS snapshot.....
2014-12-12 11:05:10: I Start/stop MASTER to check consistency.
2014-12-12 11:05:42: I Checking if the source database MASTER is stored on an ACFS snapshot
acfsutil snap create: ACFS-03048: Snapshot operation could not complete.
acfsutil snap create: ACFS-03174: The Oracle ASM Dynamic Volume Manager (Oracle ADVM) compatibility
attribute for the disk group is less than 11.2.0.3.0.
2014-12-12 11:05:42: E Error getting ACFS snapshot
This can be fixed quite easily provided that you don’t break anything. Remember that compatibility can be raised but never lowered. With everything in place, the clone is quick:
[root@server5 ~]# ./gDBClone snap -sdbname master -tdbname clone1 2014-12-12 11:20:17: I Getting host info... 2014-12-12 11:20:17: I Starting..... 2014-12-12 11:20:17: I Validating environment..... Enter the source MASTER SYSDBA password: 2014-12-12 11:20:19: I Getting OH version... 2014-12-12 11:20:27: I Checking SCAN listener server5.example.com:1521... 2014-12-12 11:20:27: I Checking database CLONE1 existence... 2014-12-12 11:20:28: I Checking registered instance CLONE1 ... 2014-12-12 11:20:31: I Checking if the Source Database MASTER it's on ASM 2014-12-12 11:20:31: I Source Database MASTER it's on ACFS 2014-12-12 11:20:31: I Checking snapshot CLONE1 existence 2014-12-12 11:20:31: I Setting up clone environment.... 2014-12-12 11:20:31: I Starting auxiliary listener.... 2014-12-12 11:21:31: I Creating ACFS snapshot..... 2014-12-12 11:21:31: I Start/stop MASTER to check consistency. 2014-12-12 11:21:58: I Checking if the source database MASTER is stored on an ACFS snapshot 2014-12-12 11:21:58: I Setting up snapshot database..... 2014-12-12 11:21:58: I Creating Clone parameter files 2014-12-12 11:22:01: I Activating clone database..... PRKO-2002 : Invalid command line option: -j 2014-12-12 11:23:04: I Successfully created clone "CLONE1" database
Now what does this database look like? I remember a conversation with Andy Colvin about this since he saw it on our ODA first, but consider this:
[oracle@server5 ~]$ sq SQL*Plus: Release 12.1.0.2.0 Production on Fri Dec 12 11:51:22 2014 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options SQL> select name from v$datafile; NAME -------------------------------------------------------------------------------- /acfs/.ACFS/snaps/CLONE1/MASTER/MASTER/datafile/o1_mf_system_b8p3pxs7_.dbf /acfs/.ACFS/snaps/CLONE1/MASTER/MASTER/datafile/o1_mf_sysaux_b8p3pz4f_.dbf /acfs/.ACFS/snaps/CLONE1/MASTER/MASTER/datafile/o1_mf_undotbs1_b8p3q2q0_.dbf /acfs/.ACFS/snaps/CLONE1/MASTER/MASTER/datafile/o1_mf_users_b8p3qo2o_.dbf
Now the interesting thing is when you look into that top-level ACFS directory:
SQL> !ls -la /acfs/ total 76 drwxr-xr-x. 5 oracle dba 4096 Dec 12 10:35 . dr-xr-xr-x. 27 root root 4096 Dec 12 10:30 .. drwx------. 2 oracle dba 65536 Dec 12 10:31 lost+found drwxr-xr-x. 3 oracle oinstall 4096 Dec 12 10:52 MASTER
No .ACFS file! Even more interestingly, you can actually see what’s in the .ACFS directory when referring to it directly
SQL> !ls -l /acfs/.ACFS total 8 drwxrwx---. 6 root root 4096 Dec 12 10:31 repl drwxr-xr-x. 3 root root 4096 Dec 12 11:21 snaps
What I want to do next is to run a few performance benchmarks on a database in ACFS to see how it holds up. But that’s for another day…
Martins Blog 
Responses
Hi Martin, if your lab is still active, can you check if the DBID has been changed in the cloned DB? I cannot see any call to `nid` inside the script.
Thank you for the post BTW, cool as always :-)
Ludo,
Yes, the DBID changes. NID is not called but the rman duplicate creates a new controlfile which generates a new DBID.
Hi Seth, I wrote “cloned” but I was actually meaning “snaped”. The “Create controlfile” by itself does change the name but not the dbid.
I’ll try it “very” soon and let you know if I’m wrong or not.
Cheers
Reblogged this on The DBSArchitect Blog.