Martins Blog

Trying to explain complex things in simple terms

Archive for the ‘Uncategorized’ Category

Creating a RAC 12.1 Data Guard Physical Standby environment (4)

Posted by Martin Bach on February 15, 2017

In the previous three parts of this series a lot of preparation work, needed for the configuration of Data Guard, was performed. In this part of the mini-series they all come to fruition. Using the Data Guard broker a switchover operation will be performed. A couple of new features in 12c make this easier. According to the “Changes in This Release for Oracle Data Guard Concepts and Administration” chapter of the 12.1 Data Guard Concepts and Administration guide:

When [you, ed.] perform a switchover from an Oracle RAC primary database to a physical standby database, it is no longer necessary to shut down all but one primary database instance.

I have always wanted to test that in a quiet moment…

I have previously blogged about another useful change that should make my life easier: the static registration of the *_DGMGRL services in the listener.ora file is no longer needed. Have a look at my Data Guard Broker Setup Changes post for more details and reference to the documentation.

NOTE: As always, this is just a demonstration using VMs in my lab, based on my notes. Your system is most likely different, so in real-life you might take a different approach. The techniques I am using here were suitable for me, and my own small scale testing. I tried to make sure they are valid, but you may want to allocate more resources in your environment. Test, test, test on your own environment on test kit first!

Now let’s get to it.

Step 1: Check the status of the configuration

In the first step I always check the configuration and make sure I can switch over. Data Guard 12c has a nifty automatic check that helps, but I always have a list of tasks I perform prior to a switchover (not shown in this blog post).

The following commands are somewhat sensitive to availability of the network – you should protect your sessions against any type of network failure! I am using screen (1) for that purpose, there are other tools out there doing similar things. Network glitches are too common to ignore, and I have come to appreciate the ability to resume work without too many problems after having seen the dreaded “broken pipe” message in my terminal window…

[oracle@rac12sec1 ~]$ dgmgrl
DGMGRL for Linux: Version - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@ncdbb
Connected as SYSDBA.
DGMGRL> show configuration

Configuration - ractest

  Protection Mode: MaxPerformance
  NCDBA - Primary database
    NCDBB - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 55 seconds ago)

DGMGRL> validate database 'NCDBB'

The command to check for switchover readiness is new to 12c as well and called “validate database”. I don’t have screen output from the situation at this point-just take my word that I was ready :) Don’t switch over if you have any concerns or doubts the operation might not succeed! “Validate database” does not relieve you from your duties to check for switchover readiness – follow your procedures.

Step 2: Switch Over

Finally, the big moment has come! It takes just one line to perform the switchover:

DGMGRL> switchover to 'NCDBB'
Performing switchover NOW, please wait...
New primary database "NCDBB" is opening...
Oracle Clusterware is restarting database "NCDBA" ...
Switchover succeeded, new primary is "NCDBB"

DGMGRL> show database 'NCDBA';

Database - NCDBA

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          0 seconds (computed 1 second ago)
  Average Apply Rate: 9.00 KByte/s
  Real Time Query:    ON
    NCDBA2 (apply instance)

Database Status:

DGMGRL> show database 'NCDBB';

Database - NCDBB

  Role:               PRIMARY
  Intended State:     TRANSPORT-ON

Database Status:


Well that was easy! Did you notice Data Guard Broker telling us that ‘Oracle Clusterware is restarting database “NCDBA” …’ ? I like it.

If you get stuck at this point something has gone wrong with the database registration in the OCR. You shouldn’t run into problems though, because you tested every aspect of the RAC system before handing the system over to its intended users, didn’t you?

Validating the new standby database shows no issues. I haven’t noticed it before but “validate database” allows you to get more verbose output:

DGMGRL> validate database verbose 'NCDBA';

  Database Role:     Physical standby database
  Primary Database:  NCDBB

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Capacity Information:
    Database  Instances        Threads
    NCDBB     2                2
    NCDBA     2                2

  Temporary Tablespace File Information:
    NCDBB TEMP Files:  1
    NCDBA TEMP Files:  1

  Flashback Database Status:
    NCDBB:  On
    NCDBA:  Off

  Data file Online Move in Progress:
    NCDBB:  No
    NCDBA:  No

  Standby Apply-Related Information:
    Apply State:      Running
    Apply Lag:        0 seconds (computed 1 second ago)
    Apply Delay:      0 minutes

  Transport-Related Information:
    Transport On:      Yes
    Gap Status:        No Gap
    Transport Lag:     0 seconds (computed 1 second ago)
    Transport Status:  Success

  Log Files Cleared:
    NCDBB Standby Redo Log Files:  Cleared
    NCDBA Online Redo Log Files:   Cleared
    NCDBA Standby Redo Log Files:  Available

  Current Log File Groups Configuration:
    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              (NCDBB)                 (NCDBA)
    1         2                       3                       Sufficient SRLs
    2         2                       3                       Sufficient SRLs

  Future Log File Groups Configuration:
    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              (NCDBA)                 (NCDBB)
    1         2                       3                       Sufficient SRLs
    2         2                       3                       Sufficient SRLs

  Current Configuration Log File Sizes:
    Thread #   Smallest Online Redo      Smallest Standby Redo
               Log File Size             Log File Size
               (NCDBB)                   (NCDBA)
    1          50 MBytes                 50 MBytes
    2          50 MBytes                 50 MBytes

  Future Configuration Log File Sizes:
    Thread #   Smallest Online Redo      Smallest Standby Redo
               Log File Size             Log File Size
               (NCDBA)                   (NCDBB)
    1          50 MBytes                 50 MBytes
    2          50 MBytes                 50 MBytes

  Apply-Related Property Settings:
    Property                        NCDBB Value              NCDBA Value
    DelayMins                       0                        0
    ApplyParallel                   AUTO                     AUTO

  Transport-Related Property Settings:
    Property                        NCDBB Value              NCDBA Value
    LogXptMode                      ASYNC                    ASYNC
    RedoRoutes                      <empty>                  <empty>
    Dependency                      <empty>                  <empty>
    DelayMins                       0                        0
    Binding                         optional                 optional
    MaxFailure                      0                        0
    MaxConnections                  1                        1
    ReopenSecs                      300                      300
    NetTimeout                      30                       30
    RedoCompression                 DISABLE                  DISABLE
    LogShipping                     ON                       ON

  Automatic Diagnostic Repository Errors:
    Error                       NCDBB    NCDBA
    No logging operation        NO       NO
    Control file corruptions    NO       NO
    SRL Group Unavailable       NO       NO
    System data file missing    NO       NO
    System data file corrupted  NO       NO
    System data file offline    NO       NO
    User data file missing      NO       NO
    User data file corrupted    NO       NO
    User data file offline      NO       NO
    Block Corruptions found     NO       NO


Isn’t that cool? That’s more information at my fingertips than I can shake a stick at! It’s also a lot more than I could think of (eg online datafile move!).

Interestingly the Broker reports that I have “Sufficient SRLs”. I have seen it complain about the number of Standby Redo Logs in the past and blogged about this Interesting observation about standby redo logs in Data Guard


After 4 (!) posts about the matter I have finally been able to perform a switchover operation. Role reversals are a much neglected operation a DBA should be comfortable with. In a crisis situation everyone needs to be clear about what needs to be done to restore service to the users. The database is usually the easier part … Success of Data Guard switchover operations also depends on the quality of change management: it is easy to “forget” applying configuration changes on the DR site.

In today’s busy times only few of us are lucky enough to intimately know each and every database we look after. What’s more common (sadly!) is that a DBA looks after 42 or more databases. This really only works without too many issues if procedures and standards are rock solid, and enforced.

Posted in 12c Release 1, Data Guard, KVM, Linux, Uncategorized | Leave a Comment »

Signed on the dotted line-Enkitec!

Posted by Martin Bach on March 20, 2013

I had to think of ‏@OyvindIsene, a great ambassador of the Norwegian Oracle User Group when I typed the heading for this post. Unlike him I have not actively been looking for new challenges but sometimes things just develop, and in my case that was a great turn of events. I am very happy to have signed on the dotted line and in a couple of weeks will join Enkitec in Europe.

How did that happen? During an Oracle conference I met Andy Colvin together with some of his colleagues during a break in the busy schedule. I already knew and respected Enkitec as a great company with lots of seriously experienced DBAs. I feel fortunate to actually know some of them already from email and other social media exchanges.

Andy and I have exchanged a few tweets in the past and I really like his blog so I was curious to meet him in person. I haven’t yet had the opportunity to go and speak at an American conference so anytime someone I know from the other side of the Atlantic comes to Europe I try to meet up. I had a great time but unfortunately had to run since my talk started a few minutes later. It was quite funny actually although I’m not so sure if my presentation was up to my own expectations. The conversations I had made a lasting impression on me.

Over the cause of the next months we remained in contact, and I had the great pleasure to meet Kerry Osborne together with Andy a little later and that was when I seriously thought that I wanted to join a team I do admire. Now with all the paperwork done and dusted, and having signed I can’t wait to get started.

Posted in Uncategorized | 6 Comments »

Lights-out management console on Supermicro boards

Posted by Martin Bach on August 2, 2012

So this is slightly off topic, as it doesn’t deal with anything directly related to Oracle, it’s more a reference to those who are using Supermicro boards like I do. The nice thing about professional hardware is that you do not need a keyboard or mouse, or even a monitor. The board I have comes with an IPMI interface, and has a KVM over IP ability. The interface can be accessed from a dedicated software or a web browser.


The suitable application to access your KVM is called IPMIView, and it’s available for MacOS, Linux and Windows.

Since I didn’t want to install a lot of application on my laptop I opted to install IPMIView for Linux on my Ubuntu 12.04 LTS desktop running within a virtual machine. However I couldn’t get it to install at all until I found a great source how to do so. Here are the steps:

  1. Download IPMIView from the Supermicro website:
  2. Make the file executable: chmod u+x IPMIView…bin
  3. Execute the file (note it requires an X11 DISPLAY): ./IPMIView…bin

It will then open an installer wizard that guides you through the installation. Although JVM based, you don’t need to download a JRE beforehand. It comes with its own-just be sure not to use it for anything but IPMIView, it seems to be an old one: 1.6.03.

That’s it-change directory to where IPMIView has been installed and launch it. Bizarrely that fails unless you are root on my system!

To connect to your server, you obviously need a bridge network interface in your virtual machine. In VMware workstation you might have to change the bridged network definition and ensure that it is bridging to the LAN port, not WLAN as it did in my case.

If you are unsure which IP you assigned to your IPMI interface you can instruct the software to perform a scan which is quite useful. Once discovered, you can save your server(s) to the interface and connect. On initial contact, the default username is ADMIN, the default password (you are guessing it …) ADMIN in upper case.

I suggest you change the password RIGHT NOW (click on the USERS tab and do so). Here’s the one of the reasons I really like the IPMIView application:

OK it’s not that much of an advantage right now as I can easily reach under my desk but once the server is out of reach then the option to reboot is nice. Another great tool is the graphical KVM console! I can even use “virtual storage”, for example an ISO image on my laptop and present this as if it was inserted in the DVD drive. Even if there is no physical DVD drive at all :) There was a glitch with the software that prevented me from selecting installation media (I haven’t tried the Windows version of IPMIView) in the virtual media menu on my Linux VM. I could see from the manual that there should be an additional tab at the bottom, next to the KVM Console but for some reason it was greyed out. There might be a way to get it back but I didn’t explore it further-the virtual media worked with the web interface.

Web browser

This looks like the way to go: simply point your web browser to the IP address of the IPMI card and off you go! For the KVM to work you need a Java runtime on your host. To be fair, the web interface is a lot nicer than the Java client, especially since it allows you to resize the window and the UI elements will scale accordingly. When executed on Windows you finally have the opportunity to mount an ISO image to the server. A windows share (I assume Samba works as well) with the ISOs is required to boot, and it’s very simple to set up and mount.


Posted in Uncategorized | Tagged: , | 2 Comments »

OT: I want this (WordPress) theme

Posted by Martin Bach on May 15, 2012

Apologies for a completely non-technical blog item – feel free to skip!

However, if you have used a Mac SE like I have, running on System 6, then you’d love this WordPress theme I just discovered:

System 6 theme

Unfortunately I can’t use it as most of the source code I’m posting is too wide.

I can almost hear the characteristic SuperDrive noise when reading one of my floppy disks. A moment of nostalgia.

Posted in Uncategorized | Tagged: , , | Leave a Comment »

Installing OEM 12c agents in RPM format

Posted by Martin Bach on November 22, 2011

One of the questions I have always asked myself revolved around: “why doesn’t Oracle package certain software as an RPM on Linux?” Well this question has recently been answered in the form of the Oracle 12c agent. It IS possible to use an RPM based installation, although it doesn’t make 100 use of RPM. I have written this post to give you an idea what happens.

The procedure is described in the OEM 12 Cloud Control Advanced Installation and Configuration Guide, chapter 6. The process is very similar to the non-RPM based agent deployment. Let’s have a loot at it in detail.

Read the rest of this entry »

Posted in Cloud Control, Uncategorized | 2 Comments »

Simplified GNS setup for RAC and newer

Posted by Martin Bach on November 17, 2011

One of the main problems I have seen with GNS (Grid Naming Service) installations was that you couldn’t really see if your DNS and DHCP configuration was correct until it’s been too late. This has been addressed, but it’s little known. There are a number of checks you can run before starting Oracle Universal Installer, and this post is about them.

What is the Grid Naming System?

I was initially drawn towards the GNS when it was initially released with It is aimed at environments where the Oracle DBAs take on (yet another) piece of work, namely the DNS administration. By virtue of “subdomain delegation”, the master DNS server responsible for “” hands off requests for a subdomain to this – – to an Oracle managed process. This was quite poorly documented initially, prompting me to figure it out myself in an earlier post:

The problem with GNS in was that you couldn’t really test if the DNS setup was sufficient for Oracle Installer to work, and I had a few attempts at the installation (the discussion here takes into account that I might not have been able to perform sufficient checking!)

Read the rest of this entry »

Posted in Uncategorized | 3 Comments »

Nice surprise-automatic statistics email

Posted by Martin Bach on January 2, 2011

Today I got a nice surprise email from WordPress. Seems like my blog wasn’t doing too badly last year. I would like to thank all visitors for passing by and reading my ramblings about this and that-mainly Oracle related.

Have a happy New Year all!

Posted in Uncategorized | Leave a Comment »

Content theft on my blog

Posted by Martin Bach on August 18, 2010

I have received an email today from Don Burleson who informed me that a number of my articles have mysteriously appeared on another weblog hosted by wordpress on the name of  For instance:

My post: has reappeared as Even the host names are identical! The next one is a bit better as he managed to remove my “” example with “”: is my but he forgot to add the corrections I made. I recognize my DNS server, auxOEL5 :)

The story goes on.

Copy: from original:

Copy: Original:

Copy: Original:

And many more. It seems my blog is replicated almost 1:1 to his.

I am very disappointed to see that someone else has to resort to such extreme measures to boost his profile. Copyright aside, this is very unprofessional and I would have expected something else. If you are interested in the whole story, have a look at

Oh, and it’s not only me, more established authors such as Alex Gorbachev, Mark Bobak and Alex Fatkulin are also victims of this.

So, and that’s it from me about this matter.


Having complained to WordPress and leaving comments on my articles they mysteriously disappeared overnight. Everyone affected should do the same, and I urge you to file complaints to wordpress to have the blog removed.

Posted in Uncategorized | 1 Comment »

It has gone really quiet!

Posted by Martin Bach on August 4, 2010

Yes, it has, and I’m really sorry but at the moment all the blog writing has to take a lower priority. I am working very hard to finish my chapters of the upcoming  Pro Oracle Database 11g RAC on Linux, which turned out to be a lot more work than I expected. The last chapter I have to write from scratch is about ASM, and it’s again a meaty one. Luckily the first draft is complete and I sent it to Apress for review and the usual editorial work.

I so decided to quit my daytime job to focus entirely on the book, which I’d like to see released between Open World and the UKOUG conference this year.

Once completed, I intend to become a freelance Oracle database consultant, specialising on RAC (no surprises here!) and Oracle 11.2 upgrades. I am available from mid September (the first 4 weeks are already booked!) if you like-just contact me!

Posted in Uncategorized | 4 Comments »

Applying PSU to RDBMS home

Posted by Martin Bach on April 22, 2010

This is a quick post with some information about me deploying PSU to my Oracle installation on ocfs2, RHEL 5.4 64bit. First of all it should be said that this is for the RDBMS, not Grid Infrastructure home. At the time of this writing I didn’t find any known issues, but you might want to check Note 1061294.1 Oracle Database Patch Set Update Known Issues before proceeding. Also, never do this in production before having tested the patch in DEV, QA and all the other environments you might have.

I started by downloading the patch-if MOS is available, you can locate it on the patches and updates tab, simple search for oracle products. Enter 9352237 as the patch number, double-check your platform and download the zipfile. I extracted the file to /tmp/9352237.

As always, it’s a good idea to read the readme.html file. First thing to notice: OPatch 11.2 is recommended, patch 6880880. Opatch is avaialble for many versions, make sure to download the one for 11.2 and your platform. Deployment is quite simple-copy the patch file to $ORACLE_HOME and unzip it. Read the rest of this entry »

Posted in 11g Release 2, Linux, Uncategorized | 12 Comments »