Inside a RAC 12c GNS cluster
Posted by Martin Bach on November 26, 2013
Based on some reader feedback I started looking at GNS again, but this time it will be for RAC 12c. According to the documentation GNS has been enhanced so that you can use it without subdomain delegation. I decided to try the “old fashioned” way though: DHCP for VIPs, SCAN IPs, subdomain delegation and the like as it is the most complex setup. I occasionally like complex.
The network setup is exactly the same as I used before in 11.2 and thankfully didn’t require any changes. The cluster I am building is a 2 node system on Oracle Linux 6.4 and the Red Hat compatible kernel. I have to use this as the Unbreakable Kernel doesn’t know about block devices made available to it via virtio-scsi. I use virtio-scsi for shared block devices very much in the same way I did for Xen.
The installation was rather rough, I had a few niggles around the DNS setup with reverse name resolution. I wish I had an updated checkip.sh available for the standard RAC installation, it would make it so much easier to detect problems in advance!
Once the networking problems were sorted out I could confirm that GNS was indeed working after the installation of both nodes:
[oracle@rac12node1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rac12node1 STABLE
ONLINE ONLINE rac12node2 STABLE
ora.SYSDG.dg
ONLINE ONLINE rac12node1 STABLE
ONLINE ONLINE rac12node2 STABLE
ora.asm
ONLINE ONLINE rac12node1 Started,STABLE
ONLINE ONLINE rac12node2 Started,STABLE
ora.net1.network
ONLINE ONLINE rac12node1 STABLE
ONLINE ONLINE rac12node2 STABLE
ora.ons
ONLINE ONLINE rac12node1 STABLE
ONLINE ONLINE rac12node2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac12node2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac12node1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac12node1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac12node1 169.254.61.234 192.1
68.101.30 192.168.10
2.30,STABLE
ora.cvu
1 ONLINE ONLINE rac12node1 STABLE
ora.gns
1 ONLINE ONLINE rac12node1 STABLE
ora.gns.vip
1 ONLINE ONLINE rac12node1 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac12node1 Open,STABLE
ora.oc4j
1 ONLINE ONLINE rac12node1 STABLE
ora.rac12node1.vip
1 ONLINE ONLINE rac12node1 STABLE
ora.rac12node2.vip
1 ONLINE ONLINE rac12node2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac12node2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac12node1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac12node1 STABLE
--------------------------------------------------------------------------------
As you can see the GNS processes (GNS + GNS-VIP) are part of the upper stack, and both are avilable. If that is the case then some IP addresses must have been acquired via DHCP, and fair enough, if I check my DHCP server I can find out that addresses were lent:
... Nov 15 14:24:22 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0 Nov 15 14:24:22 aux dhcpd: Abandoning IP address 192.168.100.32: pinged before offer Nov 15 14:24:22 aux dhcpd: Wrote 0 class decls to leases file. Nov 15 14:24:22 aux dhcpd: Wrote 2 leases to leases file. Nov 15 14:24:27 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0 Nov 15 14:24:28 aux dhcpd: DHCPOFFER on 192.168.100.33 to 00:00:00:00:00:00 via eth0 Nov 15 14:24:28 aux dhcpd: DHCPREQUEST for 192.168.100.33 (192.168.100.2) from 00:00:00:00:00:00 via eth0 Nov 15 14:24:28 aux dhcpd: DHCPACK on 192.168.100.33 to 00:00:00:00:00:00 via eth0 Nov 15 14:24:46 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0 Nov 15 14:24:47 aux dhcpd: DHCPOFFER on 192.168.100.34 to 00:00:00:00:00:00 via eth0 Nov 15 14:24:47 aux dhcpd: DHCPREQUEST for 192.168.100.34 (192.168.100.2) from 00:00:00:00:00:00 via eth0 Nov 15 14:24:47 aux dhcpd: DHCPACK on 192.168.100.34 to 00:00:00:00:00:00 via eth0 Nov 15 14:24:52 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0 Nov 15 14:24:53 aux dhcpd: DHCPOFFER on 192.168.100.35 to 00:00:00:00:00:00 via eth0 Nov 15 14:24:53 aux dhcpd: DHCPREQUEST for 192.168.100.35 (192.168.100.2) from 00:00:00:00:00:00 via eth0 Nov 15 14:24:53 aux dhcpd: DHCPACK on 192.168.100.35 to 00:00:00:00:00:00 via eth0 Nov 15 14:25:11 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0 Nov 15 14:25:12 aux dhcpd: DHCPOFFER on 192.168.100.36 to 00:00:00:00:00:00 via eth0 Nov 15 14:25:12 aux dhcpd: DHCPREQUEST for 192.168.100.36 (192.168.100.2) from 00:00:00:00:00:00 via eth0 Nov 15 14:25:12 aux dhcpd: DHCPACK on 192.168.100.36 to 00:00:00:00:00:00 via eth0 ...
Further information about the network address acquisition can be found in the gns log file in $GRID_HOME/log/$(hostname -s)/gnsd/gnsd.log. After a lot of verbose messaging about its own intialisation you can find some more interesting information.
2013-11-15 14:24:16.892: [ GNS][827389472]main::clsgndGetInstanceInfo: version: 12.1.0.1.0 address "tcp://192.168.100.37:33912" port: 33912 process ID: "11081" state: "Initializing". 2013-11-15 14:24:16.892: [ GNS][827389472]main::clsgndadvAdvertise: Listening for commands on tcp://192.168.100.37:33912 port 33912.
So it’s ready to take commands. And soon thereafter they come in:
2013-11-15 14:24:21.309: [ GNS][690910976]Resolver #0::clsgndnsAddAnswer: name: "Oracle-GNS" adding Name: "Oracle-GNS" Type: SRV service: Oracle-GNS instance: rac12gns target: Oracle-GNS protocol: tcp service name: gns.example.com sub-subdomain: (NULL) port: 33912 weight: 0 priority: 0 Unique: FALSE Flags: ALLOCATED NON-PERSISTENT NO-EXPIRATION Creation: Fri Nov 15 14:24:16 2013 Expiration Fri Nov 15 14:24:31 2013 ... 2013-11-15 14:24:38.947: [ GNS][680404736]Command #1::clsgndcpHostAdvertise: advertising name "rac12node1-vip" address: "192.168.100.33" time to live (seconds): 0 lease expiration (seconds) 0. 2013-11-15 14:24:43.601: [ GNS][680404736]Command #1::clsgnctrInsertHMAC: Connection ID: length: 48 7169...f7d85556125a 2013-11-15 14:24:51.707: [ GNS][678303488]Command #2::clsgnctrCreateReceivePacket: connection version: 12.1.0.0.0 (0xc100000) 2013-11-15 14:24:51.707: [ GNS][678303488]Command #2::clsgnctrCreateReceivePacket: Connection ID: length: 48 b720f6f1ab0...30f2fe49811677520829bb71b74fbce 2013-11-15 14:24:51.708: [ GNS][678303488]Command #2::clsgndcpWait: Running command "advertise" (4933) 2013-11-15 14:24:51.708: [ GNS][678303488]Command #2::clsgndcpHostAdvertise: advertising name "rac12gns-scan1-vip" address: "192.168.100.34" time to live (seconds): 0 lease expiration (seconds) 0. 2013-11-15 14:24:57.286: [ GNS][678303488]Command #2::clsgnctrInsertHMAC: Connection ID: length: 48 77d111eb90a30...4f6da783e4e2fd033529 2013-11-15 14:24:57.288: [ GNS][680404736]Command #1::clsgnctrCreateReceivePacket: connection version: 12.1.0.0.0 (0xc100000) 2013-11-15 14:24:57.288: [ GNS][680404736]Command #1::clsgnctrCreateReceivePacket: Connection ID: length: 48 bddc...930aa35aef01bcd880ab041c0406681958f1fb3c08d8ab51 2013-11-15 14:24:57.289: [ GNS][680404736]Command #1::clsgndcpWait: Running command "advertise" (4933) 2013-11-15 14:24:57.289: [ GNS][680404736]Command #1::clsgndcpHostAdvertise: advertising name "rac12scan" address: "192.168.100.34" time to live (seconds): 0 lease expiration (seconds) 0. 2013-11-15 14:25:11.489: [ GNS][676202240]Command #3::clsgnctrCreateReceivePacket: connection version: 12.1.0.0.0 (0xc100000) 2013-11-15 14:25:11.489: [ GNS][676202240]Command #3::clsgnctrCreateReceivePacket: Connection ID: length: 48 1898551a83efa80ee41b50dd68ea...7dba2099a097 2013-11-15 14:25:11.489: [ GNS][676202240]Command #3::clsgndcpWait: Running command "advertise" (4933) 2013-11-15 14:25:11.489: [ GNS][676202240]Command #3::clsgndcpHostAdvertise: advertising name "rac12gns-scan2-vip" address: "192.168.100.35" time to live (seconds): 0 lease expiration (seconds) 0. ...
But instead of having to go over this file you can get the same information from clusterware:
[oracle@rac12node1 ~]$ srvctl config scan SCAN name: rac12scan.gns.example.com, Network: 1 Subnet IPv4: 192.168.100.0/255.255.255.0/eth0 Subnet IPv6: SCAN 0 IPv4 VIP: -/scan1-vip/192.168.100.34 SCAN name: rac12scan.gns.example.com, Network: 1 Subnet IPv4: 192.168.100.0/255.255.255.0/eth0 Subnet IPv6: SCAN 1 IPv4 VIP: -/scan2-vip/192.168.100.35 SCAN name: rac12scan.gns.example.com, Network: 1 Subnet IPv4: 192.168.100.0/255.255.255.0/eth0 Subnet IPv6: SCAN 2 IPv4 VIP: -/scan3-vip/192.168.100.36 [oracle@rac12node1 ~]$ srvctl config nodeapps Network 1 exists Subnet IPv4: 192.168.100.0/255.255.255.0/eth0, dhcp Subnet IPv6: VIP exists: network number 1, hosting node rac12node1 VIP IPv4 Address: -/rac12node1-vip/192.168.100.33 VIP IPv6 Address: VIP exists: network number 1, hosting node rac12node2 VIP IPv4 Address: -/rac12node2-vip/192.168.100.39 VIP IPv6 Address: ONS exists: Local port 6100, remote port 6200, EM port 2016 [oracle@rac12node1 ~]$ srvctl config gns -list Oracle-GNS A 192.168.100.37 Unique Flags: 0x15 rac12gns-scan1-vip A 192.168.100.34 Unique Flags: 0x1 rac12gns-scan2-vip A 192.168.100.35 Unique Flags: 0x1 rac12gns-scan3-vip A 192.168.100.36 Unique Flags: 0x1 rac12gns.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 33912 Weight: 0 Priority: 0 Flags: 0x15 rac12gns.Oracle-GNS TXT CLUSTER_NAME="rac12gns", CLUSTER_GUID="16aaef3df3d8efd6bffb1f37101d84e3", NODE_ADDRESS="192.168.100.37", SERVER_STATE="RUNNING", VERSION="12.1.0.1.0", DOMAIN="gns.example.com" Flags: 0x15 rac12node1-vip A 192.168.100.33 Unique Flags: 0x1 rac12node2-vip A 192.168.100.39 Unique Flags: 0x1 rac12scan A 192.168.100.34 Unique Flags: 0x1 rac12scan A 192.168.100.35 Unique Flags: 0x1 rac12scan A 192.168.100.36 Unique Flags: 0x1 [root@rac12node1 ~]# srvctl config gns -detail GNS is enabled. GNS is listening for DNS server requests on port 53 GNS is using port 5,353 to connect to mDNS GNS status: OK Domain served by GNS: gns.example.com GNS version: 12.1.0.1.0 Globally unique identifier of the cluster where GNS is running: 16aaef3df3d8efd6bffb1f37101d84e3 Name of the cluster where GNS is running: rac12gns Cluster type: server. GNS log level: 1. GNS listening addresses: tcp://192.168.100.37:16934. [oracle@rac12node1 ~]$ cluvfy comp gns -postcrsinst -verbose Verifying GNS integrity Checking GNS integrity... Checking if the GNS subdomain name is valid... The GNS subdomain name "gns.example.com" is a valid domain name Checking if the GNS VIP belongs to same subnet as the public network... Public network subnets "192.168.100.0, 192.168.100.0, 192.168.100.0, 192.168.100.0, 192.168.100.0" match with the GNS VIP "192.168.100.0, 192.168.100.0, 192.168.100.0, 192.168.100.0, 192.168.100.0" Checking if the GNS VIP is a valid address... GNS VIP "gns-vip.example.com" resolves to a valid IP address Checking the status of GNS VIP... Checking if FDQN names for domain "gns.example.com" are reachable GNS resolved IP addresses are reachable GNS resolved IP addresses are reachable GNS resolved IP addresses are reachable Checking status of GNS resource... Node Running? Enabled? ------------ ------------------------ ------------------------ rac12node1 yes yes rac12node2 no yes GNS resource configuration check passed Checking status of GNS VIP resource... Node Running? Enabled? ------------ ------------------------ ------------------------ rac12node1 yes yes rac12node2 no yes GNS VIP resource configuration check passed. GNS integrity check passed Verification of GNS integrity was successful.
And finally you can use nslookup to perform name resolution:
[oracle@rac12node1 ~]$ nslookup rac12scan.gns.example.com 192.168.100.37 Server: 192.168.100.37 Address: 192.168.100.37#53 Name: rac12scan.gns.example.com Address: 192.168.100.34 Name: rac12scan.gns.example.com Address: 192.168.100.36 Name: rac12scan.gns.example.com Address: 192.168.100.35
This is reflected in the GNS log file:
013-11-15 15:06:21.190: [ GNS][682505984]Resolver #3::clsgndnsProcessNameQuestion: Query received for name: "rac12scan.gns.example.com." type: "A" 2013-11-15 15:06:21.190: [ GNS][682505984]Resolver #3::clsgndnsProcessNameQuestion: Query received for name: "rac12scan.gns.example.com." type: "A" 2013-11-15 15:06:21.190: [ GNS][682505984]Resolver #3::clsgndnsAddAnswer: name: "rac12scan" adding Name: "rac12scan" Type: A 192.168.100.34 Unique: TRUE Flags: ALLOCATED 2013-11-15 15:06:21.190: [ GNS][682505984]Resolver #3::clsgndnsAddAnswer: name: "rac12scan" adding Name: "rac12scan" Type: A 192.168.100.36 Unique: TRUE Flags: ALLOCATED 2013-11-15 15:06:21.190: [ GNS][682505984]Resolver #3::clsgndnsAddAnswer: name: "rac12scan" adding Name: "rac12scan" Type: A 192.168.100.35 Unique: TRUE Flags: ALLOCATED 2013-11-15 15:06:21.190: [ GNS][682505984]Resolver #3::clsgndnsResolve: name "rac12scan.gns.example.com." Type: Addr Resolved: TRUE
So that’s Clusterware using GNS installed. A few points to notice
- The GNS VIP is on the domain, in this case gns-vip.example.com
- The public node names have to be defined in DNS or the hosts file (or both)
- Only the VIPs (node VIPs and SCAN VIPs) are assigned by DHCP
- Have a large enough range of addresses in /etc/dhcp/dhcpd.conf
- Run cluvfy stage -pre dbinst -n node1,node2 to check if the prerequisites for the RDBMS installation are met
Happy installations!
Rick Wessman said
Hi, Martin:
The need to delegate a subdomain to GNS hasn’t really changed in 12. However, if the cluster is in flex mode, GNS runs by default and is used internally by the cluster for service advertisement. If DHCP is not in use, then no subdomain is needed.
One significant change in 12 is the ability for one GNS instance to service multiple clusters. The result is that only one subdomain needs to be added to DNS rather than one subdomain per cluster.
As an aside, I would suggest using srvctl to get information about GNS rather than examining the log file. As you point out, it’s pretty dense and is really only meant for problem diagnosis.
Thanks,
Rick
Martin Bach said
Hi Rick!
Thanks for passing by, and even more so for clarifying. I just re-read the relevant section in the documentation (http://docs.oracle.com/cd/E16655_01/install.121/e17888/networks.htm#CWLIN210). Without your comments (and probably in this context only) I would have understood it again in the same way I described.
Martin
Rick Wessman said
You’re very welcome. I agree that the wording is somewhat confusing.
Rick
Interesting GNS anomaly in 12.1.0.1.2 RAC « Martins Blog said
[…] I was encountering an interesting anomaly with my 12.1.0.1.2 RAC cluster based on Grid Naming System. I have written about the setup here. […]