Exadata experience, what does that actually mean?

As an active member of the Oracle user community I really enjoy talking to delegates at user conferences and user group meetings. As such I was very lucky having had the opportunity to attend two of them recently. I have written about the OUGN spring conference in the post before this, and I also enjoyed the AIM meeting earlier in March.

One of the subjects that always seems to come up is Exadata. Many, many DBAs want to have Exadata experience, and if only to tick a box. Now Exadata means a significant investment, in other words not every company on the planet will have one. On the other hand it’s reasonably complex to administer, therefore recruiters and other HR personal are very interested in DBAs with “Exadata experience”. Now, the reason of this blog post is an open question to the readers: what do you consider as Exadata experience?

Let me explain my reasoning. I have worked on a site with Exadata. I have logged in, I produced papers about the Smart Flash Cache, cell metric statistics, smart scans etc. Does that mean I have Exadata experience? Maybe yes. But then until recently I haven’t been involved in the One Command process. I worked my way through a huge PDF which asked all sorts of questions about the Exadata configuration. I have seen the resulting XLS file which is used to drive the configuration of the Exadata database machine. Unfortunately I didn’t have the opportunity to execute the OneCommand script myself-but I’ve seen the log files. Many Exadata DBAs haven’t seen this at all, because Oracle installed their Exadata system. Do they not have Exadata experience? Thought-provoking!

I also know a little bit about Real Application Clusters. I know about workload management and other features that aren’t widely used such as Universal Connection Pools, Fast Application Notification and the other related technologies. Does that mean I know everything there is to Exadata? Not at all-I know the concepts, and I know where to look for information. I would trust only very few people who claim they know everything about Exadata.

I think the bottom line is that there are very few people on the planet who really know enough. Most others will see a piece of the puzzle, which is not necessarily their fault but a procedural problem. In the days before 10.1 the world was simple: you had storage administrators, you had OS admins, and then database administrators. You would have used block-level SAN replication, Oracle 9i single instance and Forms and Reports 6i on Citrix Metaframe. Simple. Worked! If it got really fancy, you’d see a VCS or Power HA cold failover cluster (administered by the system admin).

Now we have Automatic Storage Management. The DBA intrudes into the world of the storage admin, and that is painful for both. We also have Oracle Clusterware: all over sudden we don’t need the system administrator to define Sun Cluster, HP ServiceGuard, SFRAC etc. So the DBA again intrudes into unknown terrain. There are many more examples, including dNFS, the Grid Naming System, Cluster Time Synchronisation, Fencing (remember IPMI?)

This change in point of view doesn’t work well with the old model which still is unchanged from the “old” days. And it gets worse when introducing Exadata. Who does what? Will the DBA administer the cells and the ASM disk groups-no way, that storage admin territory. But hey it also means the storage admin has to learn ASM…. What about patching the compute nodes-you are ideally logged in as root for opatch auto. But a DBA with root access!?!? Never. And so the DBA has to be content with what he has always done in 9i and not worry too much about the other components, which are nominally in the hands of the system and storage admin. Except when it breaks :)

So much for that, this is very much a European view, readers elsewhere may have different experience. Do YOU have Exadata experience?

Bottom Line

This article has caused more interest than I assumed it would, so maybe a summary is in order

  1. Dear recruiter, when I’m stating I have Exadata experience, please don’t expect me to be able to answer each and every answer. Depending on what I did I simply didn’t have exposure to xyz.
  2. Companies implementing Exadata should try to embrace the change.

I’m in no way critical of Exadata, in fact I really enjoy it! Infiniband, fast processors, Real Application Cluster, Data Guard… all the right ingredients to keep me happy.


19 thoughts on “Exadata experience, what does that actually mean?

  1. jarneil

    Hi Martin,

    I’m going to blow my trumpet – I think I’ve actually executed all those Exadata specific processes! No Oracle ACS getting all the fun here! So I’m *definitely* claiming Exadata experience. I don’t think this is necessarily the hard part of using exadata!

    You’ve not mentioned migration – also quite a crucial feature of any exadata project. Then there is the knowledge of the database actually sitting on Exadata this is also really where the rubber hits the road.

    The last paragraph – i disagree with. The BEST way of managing exadata is having it all within the one team, I realise this does make Exadata somewhat disruptive technology and goes against the grain of large organisations – but a lot of organisations are adopting the approach of a specific team to manage exadata.



    1. Martin Bach Post author

      I had you in mind when writing! However do you do performance monitoring and problem fixing ;) I would think that Exadata is sufficiently unique in what it offers over and above RAC (smart scan, HCC, …) that Exadata performance tuning is 75-85 standard RAC related tuning with the rest Exadata specific.

  2. kevinclosson

    This is an excellent post, Martin. It’s nearing the point where an Oracle DBA cannot do everything that is expected. The acronym no longer really fits. Database Administrator? What does that have to do with cellsrv? DBA? What does that have to do with clusters (at the platform level [you mention fencing]), etc, etc, etc.

    It’s nearing time that there should be two roles:

    1) The Oracle Platform Administrator (OPA)
    2) The Oracle Database Administrator

    The DBA tends to the *contents* of the database and applications’ needs for data from the database, tuning, etc. The OPA deals with all the activity that was once the domain of the System Admin and Storage Admin and, moreover, the OPA deals with the crushing load of maintaining Oracle software (patching).

    I can’t see it any other way. Oracle has moved their technology stack in a direction that essentially warrants this split in responsibilities. Think of any other RDBMS product where the DBA is expected to be a platform administrator as well. I’m not saying good or bad on that matter, but it is high time to consider the split.


    PS. Now how in the world are your readers going to get up to speed on Exadata without a link to my blog in your blogroll? Blush.

    1. Martin Bach Post author

      Yes, I agree. And funny enough with Exadata I see this as the case already. Where Oracle ACS is still on site, they are the OPAs (btw that sounds funny to German ears). Whereas the completely legit assumption is that they have the ties into the support organisation to deal with the machinery underneath the business critical application.

      It really is the application we should care about most, not the mechanics underneath. Sadly very few applications allow this transparent view (and I’m explicitly not leaving the Oracle universe here).

      Should we create a new job title then?

      1. Andy Colvin (@acolvin)

        I think it really depends on how an organization is set up before Exadata comes in that determines what the boundaries will be after the box is dropped off. In cases where the sysadmins and DBAs have defined roles and actually talk and share information, they continue to behave the same and the Exadata is managed by both entities, with the DBAs taking on most of the work.

        In environments where the DBAs and sysadmins are separate and rarely communicate usually end up in one of 2 situations – the sysadmins try to keep the DBAs away from their territory, or they completely wash their hands of the Exadata and leave it up to the DBAs to manage. The first usually ends up messy, and the second can as well if the DBA staff doesn’t have the knowledge (or the motivation) to manage the stack. Adding to the problem is that the sysadmins aren’t usually brought in to the discussion until after ACS (or a good Exadata partner) is finished running the configuration steps. It’s things like this that make people want to treat it like a black box.

        For me personally, it’s interesting because I get to play with the entire stack. On most traditional environments, I don’t have access to look at the storage, or much other than the database itself. With that said, it is becoming harder to just add a blanket “I know Exadata” to a resume, since the scope of Exadata is so broad. I’d be very wary of anyone who says they know everything about Exadata.

    1. kevinclosson

      @Justin : Not sure I (in particular) need to see it but I think it’s a bit absurd to have an article with deep dive covering commands documented in documentation one cannot obtain without buying Exadata. If the documentation is not open (like all other Oracle doc that I know of) why am I reading a deep-dive into these commands? This comment is not a reflection on Arup, but the closed policy of the docs instead.

    2. kevinclosson

      @Justin : Actually I have another comment about the articles you link to and in specific this article: http://www.oracle.com/technetwork/articles/oem/exadata-commands-part1-402441.html were the following quoted information is offered:

      “An index scan will look into index blocks first and then the table blocks – so, Smart Scan is not used.

      As the technical reviewer of the only two published books about Exadata and the former Performance Architect of Exadata I *would* offer my services as reviewer of these sorts of OTN articles but I’m afraid the perception exists that I cannot be both truthful about Exadata technology an un-biased at the same time. In fact, I can, and I am. Oracle sadly wouldn’t believe that though.

      So the problem with that quoted string is that the referring page (your referenced URL) states: “If you are new to Exadata, this installment is definitely for you.” We are all only human so I understand mistakes. See, the quoted string above is actually *not* true. It is, in fact, index range scans that cannot be offloaded to storage. Index full scans are most certainly offloadable.

      Readers of my blog would know this. Readers of Expert Oracle Exadata (Apress) would know this.

      If one is new to Exadata I feel it is insanely crucial that the best (correct) information be sought out. This is not child’s play and there are large amounts of money at stake. Being uninformed is as bad as being informed of incorrect information. Because of the high stakes I would rather people seek independent information about Exadata first and then see what Oracle has to say about it.

      PS. I’m quite certain Arup knows enough about Exadata to know this is a mistake. I do *not* take arms against fellow Oaktable Network members.

    3. Martin Bach Post author

      Hi Justin, thanks for reading this post!

      I have of course read them, and I am also proud owner of Expert Oracle Exadata (thanks to the authors!). Now if only I could play with these concepts at home (hint, hint).


  3. Kellyn Pot'Vin (@DBAKevlar)

    I spent my entire tenure at a company on an Exadata, pretty much day in and day out, yet I feel like I hardly scratched the surface of this architecture and software solution. I’ve done performance tuning, bundle patches and upgrades (not a real fan of the patch manager..sigh…) I didn’t really post anything on my blog about it even though I highly enjoyed the new performance tuning challenges involved with storage indexes, smart scans, bloom filters, etc. I just didn’t have anything to add after learning much of what I know from those that had already blogged about it. They were the experts, I was the student…

    This was verified when I recently moved over to Enkitec, working with the likes of Kerry Osborne, Tanel Poder, Andy Colvin and others- I hear the old, TV show Hogan’s Heroes character, Schultz’s voice in my head saying, “I know NOTHING!” :)

    There are a number of folks out there that I’ve come across that claim they possess Exadata expertise and yet know less than I do. They often claim that Exadata is the answer for EVERY Oracle environment and often make false claims. I see this no different than those that make false claims about RAC being the GOD-ALL answer for high availability…..:)

    I just need to keep on the good path and keep learning…

  4. Paul

    Hi Martin,

    Great post and lots of good follow-up.

    Working for Oracle ACS I thought it would be useful to put forward my perspective (and no we don’t get “all the fun”! :) ….

    I have Exadata “experience” in so much that I’ve installed several of them, I’m therefore used to OneCommand (having debugged it a couple of times!), Exadata patching as a whole, bare metal provisioning and the ever problematic networking requirements for installing an Exadata, which is the most common cause of issues with the installation.

    What I have only a small amount of experience of is “in life” Exadata … because most of the customers I have dealt with so far are still firmly in the project phase, considering how they might get their data warehouse onto Exadata ior testing with full data volumes. Although having experience of the configuration process is really useful I’m very keen to see more of these things in full swing – I don’t expect to have to wait too long as the numbers we are deploying mean that sooner or later there will be lots of live Exadata’s to enjoy!

    In terms of how to support an Exadata from a role and responsibility perspective, from what I know internally we will be recommending customers adopt the following approach:

    Initially form an Extended DBA team, this team would be centred around the DBA team but they would have access to specialists from the “old” organisation to help with areas that they have yet to gain the right level of skills e.g. networking, Linux Admin. This would seem to require the least amount of disruption in the short term, in the medium to long term transition to a full Database Machine Admin (DBMA) Team with dedicated resources who understand the full stack, this should be possible as the need for the extended specialists diminishes over time.



  5. Neil Chandler


    I have no Exadata experience. However, I’ve been an Oracle DBA for over 20 years. I have administered several types of *nix and Windows clusters (from setup to Production), as well as EMC’s, HP and IBM SANs. I probably know more about SAN admin than most SAN admins I have met. Yet I have no realistic chance of going into a client as an Exadata person despite being more qualified than most DBA’s (certainly all of the DBA’s I have interviewed recenty) to pick it up and understand it. I love irony. :-)



  6. Rui Amaral

    Hi Martin,

    I can honestly say I have had some experience on Exadata (and RAC, and so on) but like you I would not consider myself and expert on it or RAC or whatever. It always appears that there is more that keeps on popping up that makes me go “hmmmmmm… okay no that is something I wished I knew earlier”. I have to wonder about most of those who make such claims to expertise or about companies who demand expertise of their personnel or applicants but ultimately rarely use what they ask. And I absolutely hate the term “expert” to the point that I never mention it in an interview or in general discussion.

    Anyway, it was a great read Martin.

  7. Leighton L. Nelson (@leight0nn)

    Hi Martin,
    I find this interesting as I recently interviewed a DBA who claimed to have Exadata experience. I have zero Exadata experience (although I would love to) and the position doesn’t require it. The thing I found intriguing was that the candidate could only express certain concepts by referring to Exadata. For example if asked about workload management using services the reply would start something like “Well on Exadata you would xyz…”
    So now it is conceivable that DBAs could have some Exadata experience without having had a core compentency in an area such as RAC! I guess what I’m saying is if I was working in an Exadata shop then Exadata experience shouldn’t be so much of a requirement than core competency in it’s components. I’d rather have a DBA with core skills and then Exadata training.

  8. sandeshr76

    Martin , I think this space is too wide for a single person to know but I’m not sure only installing qualifies on to know the platform , since its pieces aka RAC , cellsrv and infiniband , one needs to know or at least understand how these bits work together and generally this is based on good ground concepts of what one knows about basic system architecture in the first place. Most dba’s are working on storage , networking and at the system level into more depth than ever and these lines will blur depending on what the organizational standards are. Your article is a good read.

  9. sandeshr76

    Martin , I think some good ol rounded experience working on RAC, ASM , understanding the system itself like the cells, the infiniband , perhaps seeing an install or the logs and conceptually having a good grounding on the underlying technologies is a good prerequisite. Some of the best Exadata dba’s I have worked with had good DBA concepts to go along with them to augment with the Exadata specifics for performance ,smart scans , direct access to storage , patching , maintenance . Your article is an interesting read

  10. Pingback: Get a feel for enterprise block level replication using drbd « Martins Blog

  11. Pingback: The Oracle Platform Administrator | Julian Dontcheff's Database Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s