RAC 12c enhancements: adding an additional SCAN-part 2
Posted by Martin Bach on April 28, 2014
In the first part of this article you could read how to add an additional network resource, additional VIPs and SCAN to an 12.1.0.1.2 cluster. In this part I hope to show you the next steps such as adding the SCAN listeners and other resources.
New SCAN listener
With the second SCAN added it is time to add the next set of SCAN listeners. This is really simple, and here is the code to add them:
[oracle@ron12cprinode1 ~]# srvctl add scan_listener -netnum 2 -listener dgscanlsnr
After starting the SCAN listeners on network 2, I can see they are indeed working correctly:
[oracle@ron12cprinode1 ~]$ srvctl start scan_listener -k 2 [oracle@ron12cprinode1 ~]$ srvctl config scan_listener -k 2 SCAN Listener DGSCANLSNR_SCAN1_NET2 exists. Port: TCP:1521 Registration invited nodes: Registration invited subnets: SCAN Listener DGSCANLSNR_SCAN2_NET2 exists. Port: TCP:1521 Registration invited nodes: Registration invited subnets: SCAN Listener DGSCANLSNR_SCAN3_NET2 exists. Port: TCP:1521 Registration invited nodes: Registration invited subnets: [oracle@ron12cprinode1 ~]$
This step needs to be repeated on the second cluster as well. As it’s more or less the same I didn’t repeat it here.
Now it’s getting slightly busier in the infrastructure! In between the various tasks I verified the status of Clusterware. At this point I had additional VIPs, a new SCAN, and new SCAN listeners.
crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ron12cprinode1 STABLE
ONLINE ONLINE ron12cprinode2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE ron12cprinode1 STABLE
ONLINE ONLINE ron12cprinode2 STABLE
ora.OCR.dg
ONLINE ONLINE ron12cprinode1 STABLE
ONLINE ONLINE ron12cprinode2 STABLE
ora.RECO.dg
ONLINE ONLINE ron12cprinode1 STABLE
ONLINE ONLINE ron12cprinode2 STABLE
ora.asm
ONLINE ONLINE ron12cprinode1 Started,STABLE
ONLINE ONLINE ron12cprinode2 Started,STABLE
ora.net1.network
ONLINE ONLINE ron12cprinode1 STABLE
ONLINE ONLINE ron12cprinode2 STABLE
ora.net2.network
ONLINE ONLINE ron12cprinode1 STABLE
ONLINE ONLINE ron12cprinode2 STABLE
ora.ons
ONLINE ONLINE ron12cprinode1 STABLE
ONLINE ONLINE ron12cprinode2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.DGSCANLSNR_SCAN1_NET2.lsnr
1 ONLINE ONLINE ron12cprinode2 STABLE
ora.DGSCANLSNR_SCAN2_NET2.lsnr
1 ONLINE ONLINE ron12cprinode2 STABLE
ora.DGSCANLSNR_SCAN3_NET2.lsnr
1 ONLINE ONLINE ron12cprinode2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.cvu
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.ron.db
1 ONLINE ONLINE ron12cprinode1 Open,STABLE
ora.ron.ron12c.svc
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.ron12cprinode1.vip
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.ron12cprinode1_2.vip
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.ron12cprinode2.vip
1 ONLINE ONLINE ron12cprinode2 STABLE
ora.ron12cprinode2_2.vip
1 ONLINE ONLINE ron12cprinode2 STABLE
ora.scan1.vip
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.scan1_net2.vip
1 ONLINE ONLINE ron12cprinode2 STABLE
ora.scan2.vip
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.scan2_net2.vip
1 ONLINE ONLINE ron12cprinode2 STABLE
ora.scan3.vip
1 ONLINE ONLINE ron12cprinode1 STABLE
ora.scan3_net2.vip
1 ONLINE ONLINE ron12cprinode2 STABLE
--------------------------------------------------------------------------------
Additional Node Listeners
Now I need another set of listeners on the 2nd network. You can use netca for this, but it’s far easier to add it on the command line.
I was going ahead and use port 1522 for this new set of listeners to remove some of the ambiguity in having multiple listeners. There is not a technical need to do so, since the listeners listen on different networks (IP addresses) they could all use port 1521.
Here is the code for reference:
[oracle@ron12cprinode1 ~]$ srvctl add listener -listener LISTENER_DG -netnum 2 [oracle@ron12cprinode1 ~]$ srvctl config listener Name: LISTENER Network: 1, Owner: oracle Home: <CRS home> End points: TCP:1521 Name: LISTENER_DG Network: 2, Owner: oracle Home: <CRS home> End points: TCP:1522
Who needs to start a GUI if this can be done with 1 line ;) Don’t forget to create the node listeners on the standby cluster as well.
Listener (cross) registration
One of the perks of 12c Clusterware is that you don’t need to set listener_networks manually anymore. If you haven’t heard about this particular parameter here is some background.
You have to define listener_networks to allow cross-registration of cluster-listener (SCAN!) and the node listener. Following the above example I need to ensure that the SCAN on network 1-the one created during the installation-registers with the listeners on the same network. I also need to ensure that my Data Guard SCAN listener registers with the DG node listeners. In versions prior to 12c this was done by manually setting the initialisation parameter listener_networks.
In 12c the documentation claims this happens automatically, and the ever-sceptic me didn’t want to believe this at first. But after a restart of the primary database I could see the following entries in the alert.log, these are set by Clusterware, not me:
NOTE: dependency between database ron and diskgroup resource ora.DATA.dg is established ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.100.51)(PORT=1521))' SCOPE=MEMORY SID='pri_1'; ALTER SYSTEM SET remote_listener=' ron12cpri-scan.example.com:1521' SCOPE=MEMORY SID='pri_1'; ALTER SYSTEM SET listener_networks='(( NAME=net2)(LOCAL_LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.102.51)(PORT=1522)))))','((NAME=net2)(REMOTE_LISTENER=ron12cpri-dgscan.dg.example.com:1521))' SCOPE=MEMORY SID='pri_1'; 2014-04-28 02:44:46.242000 -04:00 ALTER DATABASE MOUNT
192.168.100.51 is the IP address of ron12cprinode1-vip, and the local_listener defaults to the listener on that interface. The remote listener is also set to the recommended value automatically. The interesting bit lies in the next command, and I want to make sure it all works. 192.168.102.51 is the first VIP on the second (DG) network and you saw that ron12cpri-dgscan.dg.example.com has just been created. For those of you who remember setting listener_networks in 11.2 you will find this automation a big relief.
Did it work?
Cross registration is best checked with the listeners. There are quiet a few listeners to check (do a ps -ef | grep tns for fun on one of your cluster nodes):
- local node listener on network 1
- (any) SCAN listener on network 1
- local node listener on network 2
- (any) SCAN listener on network 2
A quick check to the listeners on the primary node reveals the following:
[oracle@ron12cprinode1 ~]$ lsnrctl status LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 28-APR-2014 02:50:25 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 12.1.0.1.0 - Production Start Date 23-APR-2014 10:55:44 Uptime 4 days 15 hr. 54 min. 41 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/12.1.0.1/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/ron12cprinode1/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.50)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.51)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for this service... Service "pdb1" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "priXDB" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron12c" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron_DGB" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... The command completed successfully [oracle@ron12cprinode1 ~]$
As was expected, the node listener “knows” about my database instance. What about the initial SCAN listeners?
[oracle@ron12cprinode1 ~]$ lsnrctl status LISTENER_SCAN1 LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 28-APR-2014 02:54:31 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))) STATUS of the LISTENER ------------------------ Alias LISTENER_SCAN1 Version TNSLSNR for Linux: Version 12.1.0.1.0 - Production Start Date 23-APR-2014 10:55:41 Uptime 4 days 15 hr. 58 min. 50 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/12.1.0.1/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/ron12cprinode1/listener_scan1/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.54)(PORT=1521))) Services Summary... Service "pdb1" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "priXDB" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron12c" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron_DGB" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... The command completed successfully [oracle@ron12cprinode1 ~]$
Looks ok too. The new Data Guard SCAN listener is next:
[oracle@ron12cprinode2 ~]$ lsnrctl status DGSCANLSNR_SCAN1_NET2 LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 28-APR-2014 02:54:54 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=DGSCANLSNR_SCAN1_NET2))) STATUS of the LISTENER ------------------------ Alias DGSCANLSNR_SCAN1_NET2 Version TNSLSNR for Linux: Version 12.1.0.1.0 - Production Start Date 28-APR-2014 02:16:46 Uptime 0 days 0 hr. 38 min. 8 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/12.1.0.1/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/ron12cprinode2/dgscanlsnr_scan1_net2/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=DGSCANLSNR_SCAN1_NET2))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.102.54)(PORT=1521))) Services Summary... Service "pdb1" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron12c" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron_DGB" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... The command completed successfully [oracle@ron12cprinode2 ~]$
And finally, what about the new LISTENER_DG, the Data Guard node listener:
[oracle@ron12cprinode1 admin]$ lsnrctl status LISTENER_DG LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 28-APR-2014 03:14:05 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_DG))) STATUS of the LISTENER ------------------------ Alias LISTENER_DG Version TNSLSNR for Linux: Version 12.1.0.1.0 - Production Start Date 23-APR-2014 10:55:41 Uptime 4 days 16 hr. 18 min. 25 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/12.1.0.1/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/ron12cprinode1/listener_dg/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_DG))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.102.51)(PORT=1522))) Services Summary... Service "RON" has 3 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "pdb1" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron12c" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... Service "ron_DGB" has 1 instance(s). Instance "pri_1", status READY, has 1 handler(s) for this service... The command completed successfully [oracle@ron12cprinode1 admin]$
So it appears the listeners have been able to register with the LREG-processes of my database. In the next part of the series I’ll add the static database registration to listener.ora and duplicate my database for use as a physical standby.
Martin Nash said
Hi Martin – I just ran into “Bug 19884033 – ORA-2097 and ORA-119 attempting to set listener_networks parameter by the agent (Doc ID 19884033.8)” which stops the automatic updating of LISTENER_NETWORKS from working on 12.1.0.1 and 12.1.0.2. I don’t see what version your databases are in the post. Were they 11.2?
Martin Nash said
I didn’t investigate yet, but maybe it is not having SCAN listeners on the second network results in the malformed LISTENER_NETWORKS parameter…
Martin Bach said
I do have SCAN listener on the second network so that’s probably not the cause. I re-tested yesterday in GI/RDBMS 12.1.0.2.5 and the agent doesn’t set the remote listener correctly :(
Johannes Ahrends said
Any idea on how to fix the bug that listener_networks will no longer be updated with PSU 5 and 6? The patch for 19884033.8 doesn’t seem to work.
Martin Bach said
I remember that I ran into this before, and I just set it manually.