Martins Blog

Trying to explain complex things in simple terms

«
»

Inside a RAC 12c GNS cluster

Posted by Martin Bach on November 26, 2013

Based on some reader feedback I started looking at GNS again, but this time it will be for RAC 12c. According to the documentation GNS has been enhanced so that you can use it without subdomain delegation. I decided to try the “old fashioned” way though: DHCP for VIPs, SCAN IPs, subdomain delegation and the like as it is the most complex setup. I occasionally like complex.

The network setup is exactly the same as I used before in 11.2 and thankfully didn’t require any changes. The cluster I am building is a 2 node system on Oracle Linux 6.4 and the Red Hat compatible kernel. I have to use this as the Unbreakable Kernel doesn’t know about block devices made available to it via virtio-scsi. I use virtio-scsi for shared block devices very much in the same way I did for Xen.

The installation was rather rough, I had a few niggles around the DNS setup with reverse name resolution. I wish I had an updated checkip.sh available for the standard RAC installation, it would make it so much easier to detect problems in advance!

Once the networking problems were sorted out I could confirm that GNS was indeed working after the installation of both nodes:

[oracle@rac12node1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac12node1               STABLE
               ONLINE  ONLINE       rac12node2               STABLE
ora.SYSDG.dg
               ONLINE  ONLINE       rac12node1               STABLE
               ONLINE  ONLINE       rac12node2               STABLE
ora.asm
               ONLINE  ONLINE       rac12node1               Started,STABLE
               ONLINE  ONLINE       rac12node2               Started,STABLE
ora.net1.network
               ONLINE  ONLINE       rac12node1               STABLE
               ONLINE  ONLINE       rac12node2               STABLE
ora.ons
               ONLINE  ONLINE       rac12node1               STABLE
               ONLINE  ONLINE       rac12node2               STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac12node2               STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rac12node1               169.254.61.234 192.1
                                                             68.101.30 192.168.10
                                                             2.30,STABLE
ora.cvu
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.gns
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.gns.vip
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rac12node1               Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.rac12node1.vip
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.rac12node2.vip
      1        ONLINE  ONLINE       rac12node2               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac12node2               STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac12node1               STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac12node1               STABLE
--------------------------------------------------------------------------------

As you can see the GNS processes (GNS + GNS-VIP) are part of the upper stack, and both are avilable. If that is the case then some IP addresses must have been acquired via DHCP, and fair enough, if I check my DHCP server I can find out that addresses were lent:

...
Nov 15 14:24:22 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0
Nov 15 14:24:22 aux dhcpd: Abandoning IP address 192.168.100.32: pinged before offer
Nov 15 14:24:22 aux dhcpd: Wrote 0 class decls to leases file.
Nov 15 14:24:22 aux dhcpd: Wrote 2 leases to leases file.
Nov 15 14:24:27 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0
Nov 15 14:24:28 aux dhcpd: DHCPOFFER on 192.168.100.33 to 00:00:00:00:00:00 via eth0
Nov 15 14:24:28 aux dhcpd: DHCPREQUEST for 192.168.100.33 (192.168.100.2) from
 00:00:00:00:00:00 via eth0
Nov 15 14:24:28 aux dhcpd: DHCPACK on 192.168.100.33 to 00:00:00:00:00:00 via eth0
Nov 15 14:24:46 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0
Nov 15 14:24:47 aux dhcpd: DHCPOFFER on 192.168.100.34 to 00:00:00:00:00:00 via eth0
Nov 15 14:24:47 aux dhcpd: DHCPREQUEST for 192.168.100.34 (192.168.100.2) from 00:00:00:00:00:00 via eth0
Nov 15 14:24:47 aux dhcpd: DHCPACK on 192.168.100.34 to 00:00:00:00:00:00 via eth0
Nov 15 14:24:52 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0
Nov 15 14:24:53 aux dhcpd: DHCPOFFER on 192.168.100.35 to 00:00:00:00:00:00 via eth0
Nov 15 14:24:53 aux dhcpd: DHCPREQUEST for 192.168.100.35 (192.168.100.2) from
00:00:00:00:00:00 via eth0
Nov 15 14:24:53 aux dhcpd: DHCPACK on 192.168.100.35 to 00:00:00:00:00:00 via eth0
Nov 15 14:25:11 aux dhcpd: DHCPDISCOVER from 00:00:00:00:00:00 via eth0
Nov 15 14:25:12 aux dhcpd: DHCPOFFER on 192.168.100.36 to 00:00:00:00:00:00 via eth0
Nov 15 14:25:12 aux dhcpd: DHCPREQUEST for 192.168.100.36 (192.168.100.2) from
00:00:00:00:00:00 via eth0
Nov 15 14:25:12 aux dhcpd: DHCPACK on 192.168.100.36 to 00:00:00:00:00:00 via eth0
...

Further information about the network address acquisition can be found in the gns log file in $GRID_HOME/log/$(hostname -s)/gnsd/gnsd.log. After a lot of verbose messaging about its own intialisation you can find some more interesting information.

2013-11-15 14:24:16.892: [     GNS][827389472]main::clsgndGetInstanceInfo: version: 12.1.0.1.0
 address "tcp://192.168.100.37:33912" port: 33912 process ID: "11081" state: "Initializing".
2013-11-15 14:24:16.892: [     GNS][827389472]main::clsgndadvAdvertise: Listening for commands
 on tcp://192.168.100.37:33912 port 33912.

So it’s ready to take commands. And soon thereafter they come in:

2013-11-15 14:24:21.309: [     GNS][690910976]Resolver #0::clsgndnsAddAnswer: name:
 "Oracle-GNS" adding Name: "Oracle-GNS" Type: SRV service: Oracle-GNS instance: rac12gns
  target: Oracle-GNS protocol: tcp service name: gns.example.com sub-subdomain: (NULL)
  port: 33912 weight: 0 priority: 0 Unique: FALSE Flags: ALLOCATED NON-PERSISTENT
  NO-EXPIRATION Creation: Fri Nov 15 14:24:16 2013  Expiration Fri Nov 15 14:24:31 2013
...
2013-11-15 14:24:38.947: [     GNS][680404736]Command #1::clsgndcpHostAdvertise:
 advertising name "rac12node1-vip" address: "192.168.100.33" time to live (seconds): 0
 lease expiration (seconds) 0.
2013-11-15 14:24:43.601: [     GNS][680404736]Command #1::clsgnctrInsertHMAC: Connection
 ID: length: 48 7169...f7d85556125a
2013-11-15 14:24:51.707: [     GNS][678303488]Command #2::clsgnctrCreateReceivePacket:
 connection version: 12.1.0.0.0 (0xc100000)
2013-11-15 14:24:51.707: [     GNS][678303488]Command #2::clsgnctrCreateReceivePacket:
 Connection ID: length: 48 b720f6f1ab0...30f2fe49811677520829bb71b74fbce
2013-11-15 14:24:51.708: [     GNS][678303488]Command #2::clsgndcpWait: Running command
 "advertise" (4933)
2013-11-15 14:24:51.708: [     GNS][678303488]Command #2::clsgndcpHostAdvertise:
 advertising name "rac12gns-scan1-vip" address: "192.168.100.34" time to
 live (seconds): 0 lease expiration (seconds) 0.
2013-11-15 14:24:57.286: [     GNS][678303488]Command #2::clsgnctrInsertHMAC: Connection
 ID: length: 48 77d111eb90a30...4f6da783e4e2fd033529
2013-11-15 14:24:57.288: [     GNS][680404736]Command #1::clsgnctrCreateReceivePacket:
  connection version: 12.1.0.0.0 (0xc100000)
2013-11-15 14:24:57.288: [     GNS][680404736]Command #1::clsgnctrCreateReceivePacket:
 Connection ID: length: 48 bddc...930aa35aef01bcd880ab041c0406681958f1fb3c08d8ab51
2013-11-15 14:24:57.289: [     GNS][680404736]Command #1::clsgndcpWait: Running command
 "advertise" (4933)
2013-11-15 14:24:57.289: [     GNS][680404736]Command #1::clsgndcpHostAdvertise:
 advertising name "rac12scan" address: "192.168.100.34" time to live (seconds): 0
 lease expiration (seconds) 0.
2013-11-15 14:25:11.489: [     GNS][676202240]Command #3::clsgnctrCreateReceivePacket:
 connection version: 12.1.0.0.0 (0xc100000)
2013-11-15 14:25:11.489: [     GNS][676202240]Command #3::clsgnctrCreateReceivePacket:
 Connection ID: length: 48 1898551a83efa80ee41b50dd68ea...7dba2099a097
2013-11-15 14:25:11.489: [     GNS][676202240]Command #3::clsgndcpWait: Running command
 "advertise" (4933)
2013-11-15 14:25:11.489: [     GNS][676202240]Command #3::clsgndcpHostAdvertise:
 advertising name "rac12gns-scan2-vip" address: "192.168.100.35" time to live (seconds):
 0 lease expiration (seconds) 0.
...

But instead of having to go over this file you can get the same information from clusterware:

[oracle@rac12node1 ~]$ srvctl config scan
SCAN name: rac12scan.gns.example.com, Network: 1
Subnet IPv4: 192.168.100.0/255.255.255.0/eth0
Subnet IPv6:
SCAN 0 IPv4 VIP: -/scan1-vip/192.168.100.34
SCAN name: rac12scan.gns.example.com, Network: 1
Subnet IPv4: 192.168.100.0/255.255.255.0/eth0
Subnet IPv6:
SCAN 1 IPv4 VIP: -/scan2-vip/192.168.100.35
SCAN name: rac12scan.gns.example.com, Network: 1
Subnet IPv4: 192.168.100.0/255.255.255.0/eth0
Subnet IPv6:
SCAN 2 IPv4 VIP: -/scan3-vip/192.168.100.36

[oracle@rac12node1 ~]$ srvctl config nodeapps
Network 1 exists
Subnet IPv4: 192.168.100.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
VIP exists: network number 1, hosting node rac12node1
VIP IPv4 Address: -/rac12node1-vip/192.168.100.33
VIP IPv6 Address:
VIP exists: network number 1, hosting node rac12node2
VIP IPv4 Address: -/rac12node2-vip/192.168.100.39
VIP IPv6 Address:
ONS exists: Local port 6100, remote port 6200, EM port 2016

[oracle@rac12node1 ~]$ srvctl config gns -list
Oracle-GNS A 192.168.100.37 Unique Flags: 0x15
rac12gns-scan1-vip A 192.168.100.34 Unique Flags: 0x1
rac12gns-scan2-vip A 192.168.100.35 Unique Flags: 0x1
rac12gns-scan3-vip A 192.168.100.36 Unique Flags: 0x1
rac12gns.Oracle-GNS SRV Target: Oracle-GNS Protocol: tcp Port: 33912 Weight: 0 Priority: 0
  Flags: 0x15
rac12gns.Oracle-GNS TXT CLUSTER_NAME="rac12gns", CLUSTER_GUID="16aaef3df3d8efd6bffb1f37101d84e3",
  NODE_ADDRESS="192.168.100.37", SERVER_STATE="RUNNING", VERSION="12.1.0.1.0",
  DOMAIN="gns.example.com" Flags: 0x15
rac12node1-vip A 192.168.100.33 Unique Flags: 0x1
rac12node2-vip A 192.168.100.39 Unique Flags: 0x1
rac12scan A 192.168.100.34 Unique Flags: 0x1
rac12scan A 192.168.100.35 Unique Flags: 0x1
rac12scan A 192.168.100.36 Unique Flags: 0x1

[root@rac12node1 ~]# srvctl config gns -detail
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5,353 to connect to mDNS
GNS status: OK
Domain served by GNS: gns.example.com
GNS version: 12.1.0.1.0
Globally unique identifier of the cluster where GNS is running: 16aaef3df3d8efd6bffb1f37101d84e3
Name of the cluster where GNS is running: rac12gns
Cluster type: server.
GNS log level: 1.
GNS listening addresses: tcp://192.168.100.37:16934.

[oracle@rac12node1 ~]$ cluvfy comp gns -postcrsinst -verbose

Verifying GNS integrity

Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "gns.example.com" is a valid domain name
Checking if the GNS VIP belongs to same subnet as the public network...
Public network subnets "192.168.100.0, 192.168.100.0, 192.168.100.0, 192.168.100.0,
  192.168.100.0" match with the GNS VIP "192.168.100.0, 192.168.100.0, 192.168.100.0,
  192.168.100.0, 192.168.100.0"
Checking if the GNS VIP is a valid address...
GNS VIP "gns-vip.example.com" resolves to a valid IP address
Checking the status of GNS VIP...
Checking if FDQN names for domain "gns.example.com" are reachable

GNS resolved IP addresses are reachable

GNS resolved IP addresses are reachable

GNS resolved IP addresses are reachable
Checking status of GNS resource...
  Node          Running?                  Enabled?
  ------------  ------------------------  ------------------------
  rac12node1    yes                       yes
  rac12node2    no                        yes

GNS resource configuration check passed
Checking status of GNS VIP resource...
  Node          Running?                  Enabled?
  ------------  ------------------------  ------------------------
  rac12node1    yes                       yes
  rac12node2    no                        yes

GNS VIP resource configuration check passed.

GNS integrity check passed

Verification of GNS integrity was successful.

And finally you can use nslookup to perform name resolution:

[oracle@rac12node1 ~]$ nslookup rac12scan.gns.example.com 192.168.100.37
Server:         192.168.100.37
Address:        192.168.100.37#53

Name:   rac12scan.gns.example.com
Address: 192.168.100.34
Name:   rac12scan.gns.example.com
Address: 192.168.100.36
Name:   rac12scan.gns.example.com
Address: 192.168.100.35

This is reflected in the GNS log file:

013-11-15 15:06:21.190: [     GNS][682505984]Resolver #3::clsgndnsProcessNameQuestion:
 Query received for name: "rac12scan.gns.example.com." type: "A"
2013-11-15 15:06:21.190: [     GNS][682505984]Resolver #3::clsgndnsProcessNameQuestion:
 Query received for name: "rac12scan.gns.example.com." type: "A"
2013-11-15 15:06:21.190: [     GNS][682505984]Resolver #3::clsgndnsAddAnswer: name:
 "rac12scan" adding Name: "rac12scan" Type: A 192.168.100.34 Unique: TRUE Flags: ALLOCATED
2013-11-15 15:06:21.190: [     GNS][682505984]Resolver #3::clsgndnsAddAnswer: name:
 "rac12scan" adding Name: "rac12scan" Type: A 192.168.100.36 Unique: TRUE Flags: ALLOCATED
2013-11-15 15:06:21.190: [     GNS][682505984]Resolver #3::clsgndnsAddAnswer: name:
 "rac12scan" adding Name: "rac12scan" Type: A 192.168.100.35 Unique: TRUE Flags: ALLOCATED
2013-11-15 15:06:21.190: [     GNS][682505984]Resolver #3::clsgndnsResolve: name
 "rac12scan.gns.example.com." Type: Addr Resolved: TRUE

So that’s Clusterware using GNS installed. A few points to notice

  • The GNS VIP is on the domain, in this case gns-vip.example.com
  • The public node names have to be defined in DNS or the hosts file (or both)
  • Only the VIPs (node VIPs and SCAN VIPs) are assigned by DHCP
  • Have a large enough range of addresses in /etc/dhcp/dhcpd.conf
  • Run cluvfy stage -pre dbinst -n node1,node2 to check if the prerequisites for the RDBMS installation are met

Happy installations!

This entry was posted on November 26, 2013 at 22:50 and is filed under 12c Release 1, KVM, Linux. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

4 Responses to “Inside a RAC 12c GNS cluster”

  1. Rick Wessman said

    November 27, 2013 at 16:33

    Hi, Martin:

    The need to delegate a subdomain to GNS hasn’t really changed in 12. However, if the cluster is in flex mode, GNS runs by default and is used internally by the cluster for service advertisement. If DHCP is not in use, then no subdomain is needed.

    One significant change in 12 is the ability for one GNS instance to service multiple clusters. The result is that only one subdomain needs to be added to DNS rather than one subdomain per cluster.

    As an aside, I would suggest using srvctl to get information about GNS rather than examining the log file. As you point out, it’s pretty dense and is really only meant for problem diagnosis.

    Thanks,
    Rick

    Reply

  2. Interesting GNS anomaly in 12.1.0.1.2 RAC « Martins Blog said

    March 13, 2014 at 14:21

    […] I was encountering an interesting anomaly with my 12.1.0.1.2 RAC cluster based on Grid Naming System. I have written about the setup here. […]

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out / Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out / Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out / Change )

Google+ photo

You are commenting using your Google+ account. ( Log Out / Change )

Cancel

Connecting to %s

«
»
 
%d bloggers like this: