Thanks to Andy Colvin (@acolvin) and the Enkitec lab I have been able to get my hands on an ODA X4-2. And since that’s a lot quieter than turning on my own lab server, and also a lot faster I used the environment to test RAC One Node in 12.1.0.2.1. I recently had a question from a customer about the usefulness of the solution and what it brings to the HA table. Well here you go.
There is just a slight caveat: to make use of the high availability features in RAC One Node you need to have an application that has been written to take advantage of them. Using Fast Connection Failover or Application Continuity is the way to go. Although I have written sample code to exploit the functionality of FCF/FAN and Application Continuity it has recently come to my attention that the very popular swingbench benchmark can do the same. I saw the connection pool tab in my demos but it didn’t “click”. Now I would like to show you what happens with your application during common scenarios in RAC One Node.
Scenario 1: Database relocation
For this blog post I’d like to try a database live migration from the first to the second host in the cluster. The database has been created using oakcli, primarily because I wanted to get more familiar with it. This is ODA release 12.1.2.1.0, the latest and greatest at the time of writing. I also love the fact that the ODA comes with quite a few of these: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz. To be precise, each node has 2s24c48t of Ivy Bridge Xeons. Nice.
[oracle@server1 ~]$ /opt/oracle/oak/bin/oakcli show version Version ------- 12.1.2.1.0
As root (indeed!) I invoked oakcli, the primary ODA command line tool. The first time around I was bitten by not having read the documentation: if you specify the -version flag oakcli will create a new RDBMS home in both cluster nodes. That was _not_ what I had in mind. On the other hand I am now familiar with checking the environment (various permutations of oakcli show…) and deleting an Oracle home. This command created the datbase in the end:
[root@server1 ~]# oakcli create database -db RON -oh OraDb12102_home1
The command will prompt you for the datbase type (single instance, RAC One, clustered) – good stuff, simple to use, and more importantly: it worked first time.
Databases in ACFS
Now remember that from ODA release 12.1.2 onwards 11.2.0.4 and 12.1.x databases will be created on ACFS. In my case that looks slightly odd, I guess I have to get used to it first:
SQL> select name from v$datafile;
NAME
---------------------------------------------------------------------------------------------
/u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/datafile/o1_mf_system_b93xdj44_.dbf
/u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/datafile/o1_mf_sysaux_b93xdmhk_.dbf
/u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/datafile/o1_mf_undotbs1_b93xdosz_.dbf
/u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/datafile/o1_mf_undotbs2_b93xdypp_.dbf
/u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/datafile/o1_mf_users_b93xdzjj_.dbf
/u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/datafile/o1_mf_soe_tbs_b93yl62m_.dbf
6 rows selected.
SQL> select name from v$tempfile;
NAME
----------------------------------------------------------------------------------------
/u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/datafile/o1_mf_temp_b93xdpl2_.tmp
SQL> select member from v$logfile
MEMBER
-----------------------------------------------------------------------------
/u01/app/oracle/oradata/datastore/RON/RON/onlinelog/o1_mf_1_b93xd3tb_.log
/u01/app/oracle/oradata/datastore/RON/RON/onlinelog/o1_mf_2_b93xd9m5_.log
/u01/app/oracle/oradata/datastore/RON/RON/onlinelog/o1_mf_3_b93y0hrq_.log
/u01/app/oracle/oradata/datastore/RON/RON/onlinelog/o1_mf_4_b93y0pf9_.log
SQL> show parameter control_files
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /u01/app/oracle/oradata/datast
ore/RON/RON/controlfile/o1_mf_
b93xd3ll_.ctl
SQL> show parameter db_recovery_file_dest
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string /u01/app/oracle/fast_recovery_
area/datastore/RON
db_recovery_file_dest_size big integer 465G
The funny bit is that /u02/app/oracle/oradata/datastore contains snapshots… This is visible in the data file location where you find an .ACFS/snaps component. I believe the naming convention is to use .ACFS (invisible) followed by the snapshot name. You can view the snapshot detail using acfsutil:
[grid@server1 ~]$ acfsutil snap info RON /u02/app/oracle/oradata/datastore/ snapshot name: RON snapshot location: /u02/app/oracle/oradata/datastore/.ACFS/snaps/RON RO snapshot or RW snapshot: RW parent name: /u02/app/oracle/oradata/datastore/ snapshot creation time: Wed Dec 17 15:26:24 2014
Interestingly the /u01 data store does not have a snapshot:
[grid@server1 ~]$ acfsutil snap info /u01/app/oracle/oradata/datastore
number of snapshots: 0
snapshot space usage: 0 ( 0.00 )
But I digress…
Database setup and configuration
After the database has been created, it is defined in Clusterware as follows:
[oracle@server1 ~]$ srvctl config database -d ron Database unique name: RON Database name: RON Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1 Oracle user: oracle Spfile: /u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/spfileRON.ora Password file: /u02/app/oracle/oradata/datastore/.ACFS/snaps/RON/RON/orapwRON Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: Disk Groups: Mount point paths: /u01/app/oracle/fast_recovery_area/datastore,/u01/app/oracle/oradata/datastore,/u02/app/oracle/oradata/datastore Services: RON_racone Type: RACOneNode Online relocation timeout: 30 Instance name prefix: RON Candidate servers: server1,server2 OSDBA group: dba OSOPER group: racoper Database instances: Database is administrator managed [oracle@server1 ~]$
The mandatory service is defined like this:
[oracle@server1 ~]$ srvctl config service -d ron Service name: RON_racone Server pool: Cardinality: 1 Disconnect: false Service role: PRIMARY Management policy: AUTOMATIC DTP transaction: false AQ HA notifications: false Global: false Commit Outcome: false Failover type: Failover method: TAF failover retries: TAF failover delay: Connection Load Balancing Goal: LONG Runtime Load Balancing Goal: NONE TAF policy specification: NONE Edition: Pluggable database name: Maximum lag time: ANY SQL Translation Profile: Retention: 86400 seconds Replay Initiation Time: 300 seconds Session State Consistency: GSM Flags: 0 Service is enabled Preferred instances: RON_2 Available instances: [oracle@server1 ~]$
With RAC One Node it is even more important to connect to the service, and not to the instance.
Getting ready for the testing
The situation before the test is shown here:
[oracle@server1 ~]$ srvctl status database -d ron Instance RON_2 is running on node server2 Online relocation: INACTIVE [oracle@server1 ~]$ srvctl status service -d ron Service RON_racone is running on instance(s) RON_2
For swingbench to benefit from FCF it needs to be configured so that it uses a connection pool. I have used the swingbench FAQ (http://www.dominicgiles.com/swingbenchfaq.html) to do so. My connection is defined as follows:
<Connection>
<UserName>soe</UserName>
<Password>soe</Password>
<ConnectString>//cluster-scan/RON_racone</ConnectString>
<DriverType>Oracle jdbc Driver</DriverType>
<ConnectionPooling>
<PooledInitialLimit>50</PooledInitialLimit>
<PooledMinLimit>25</PooledMinLimit>
<PooledMaxLimit>100</PooledMaxLimit>
<PooledInactivityTimeOut>50</PooledInactivityTimeOut>
<PooledConnectionWaitTimeout>45</PooledConnectionWaitTimeout>
<PooledPropertyCheckInterval>10</PooledPropertyCheckInterval>
<PooledAbandonedConnectionTimeout>120</PooledAbandonedConnectionTimeout>
</ConnectionPooling>
<Properties>
<Property Key="StatementCaching">50</Property>
<Property Key="FastFailover">true</Property>
<Property Key="OnsConfiguration">nodes=server1:6200,server2:6200</Property>
<Property Key="AppContinuityDriver">true</Property>
<Property Key="FetchSize">20</Property>
</Properties>
</Connection>
This part and the rest of the configuration is based largely on this post: Application Continuity in Oracle Database 12c (12.1.0.2).
Now – finally – for the test!
Starting charbench will trigger the creation of 100 sessions on the currently active instance (I set NumberOfUsers to 100) and they will be chugging along nicely. Here is the output after a couple of minutes:
[oracle@server1 bin]$ ./charbench -c oda_soe.xml Author : Dominic Giles Version : 2.5.0.952 Results will be written to results.xml. Hit Return to Terminate Run... Time Users TPM TPS 7:24:28 AM 100 17242 291
I saw about 290 TPS for a total of around 18,000 TPM over some time. Let’s start the live migration. But first, I’d like to use the glass bowl to see what might happen (new in 12.1.0.2):
[oracle@server1 ~]$ srvctl predict database -db RON -verbose Database ron will be started on node server1 Database ron will be stopped on node server2 Service ron_racone will be started on node server1
Well I guess that might be correct, but let’s try:
[oracle@server1 ~]$ srvctl relocate database -d ron -verbose -node server1 Configuration updated to two instances Instance RON_1 started Services relocated Waiting for up to 30 minutes for instance RON_2 to stop ... Instance RON_2 stopped Configuration updated to one instance [oracle@server1 ~]$
While this command executed I didn’t lose a single connection – 100 SOE connections were always established. It also takes a few second for the cache to warm up, during which the transaction rate dips a little bit:
[oracle@server1 bin]$ ./charbench -c oda_soe.xml Author : Dominic Giles Version : 2.5.0.952 Results will be written to results.xml. Hit Return to Terminate Run... Time Users TPM TPS 7:29:11 AM 100 13135 191
One minute after this the throughput is back to normal.
[oracle@server1 bin]$ ./charbench -c oda_soe.xml Author : Dominic Giles Version : 2.5.0.952 Results will be written to results.xml. Hit Return to Terminate Run... Time Users TPM TPS 7:30:50 AM 100 16800 295
Online relocation with RAC One node is certainly a possibility and works nicely if your application is ready for connection pools and the new way of connecting to the database.
Preview of the next article in the series
In the next article I’d like to add a couple of things I haven’t had time to test yet: 12.1.0.x extends the concept of the database resident connection pool to Java applications (and other middle tier) which would allow me to scale even further. I’d also like to show you what happens if the current RAC One Node instance fails. Stay tuned!
Reblogged this on The DBSArchitect Blog.