Comments for Martins Blog

Comment on Linux large pages and non-uniform memory distribution by kevinclosson

kevinclosson — Sun, 19 May 2013 17:16:26 +0000

Ah yes, the diagram you refer to reminds me of my bad habit of ignoring QPI links that cannot be used to link sockets. The E7 has 4 links but one is used to connect to the IOH so I discount it–leaving the 3 for socketsocket linking. Starting with SNB we no longer sacrifice a QPI link to attach to an IOH since multi-lane DMI 2.0 is the path to I/O.

Comment on Linux large pages and non-uniform memory distribution by Martin Bach

Martin Bach — Sat, 18 May 2013 21:37:32 +0000

Hi Kevin,

I was silently hoping for you to correct me on Westmere-EX vs Sandy Bridge comparison. Nevertheless I should have read my references right. Westmere EX is indeed shown with 4 links in the diagram, I just happened not to see it. This is what I found:

http://www.qdpma.com/systemarchitecture/systemarchitecture_qpi.html

Intel seems to have published something similar at hot chips too in HC22.24.610-Nagara-Intel-6-Westmere-EX.pdf‎ which is hosted at hotchips.org

Thanks again for clarifying.

Comment on Linux large pages and non-uniform memory distribution by kevinclosson

kevinclosson — Fri, 17 May 2013 23:52:13 +0000

> , and this is exactly why you get Westmere-EX in the X3-8, and not Sandy Bridge as in the X3-2.

The laggard state of X3-8 (remaining with Westmere EX) doesn’t really have to do with glue or no glue. It has to do with links. The reason is because there are no 3-QPI Sandy Bridge processors. All Sandy Sandy Bridge E5 SKUs have 2 QPI links which can work either to double-up Sock-Sock as in E5-2600 or to make a 4 socket system (with a hop of course). Westmere EX (aka E7) on the other hand has 3 QPI links so one can make a 4-socket box with no hops or a glue/glueless 8 way box with hops. The “glue” affects the number of hops. No glue is 2 hops. This is why you’ll *never* see Oracle publish a 4-way Westmere EX TPC-C because it would show a massive drop in scalability from 4 sockets to 8 sockets due to the hops.This fact also shows up in the TPC-C per core when comparing Oracle on X3-8 Server (The x4800 really) compared to any E5-2600 Oracle TPCC. Don’t get me wrong, 8-socket WSM-EX is a very tough thing to get right. Nobody has linear 8-socket WSM-EX with shared-data workloads.

Ivy EX will put that crucial extra QPI link back in and thus we’ll see 12TB 8-sock boxes with up to 120 cores..which begs the question, “Offload processing, why?” :-)

Comment on RAC One Node and Database Protection by TAF is not working on Rac One Node in case of node failure ? Well it depends | bdt's oracle blog

Thu, 16 May 2013 15:38:45 +0000

[…] Martin Bach did a good study of Rac One Node capability and especially of what happens with session failover during a database relocation or during a node failure into this blog post. […]

Comment on More on use_large_pages in Linux and 11.2.0.3 by Dave

Dave — Thu, 16 May 2013 12:59:29 +0000

Thanks Martin… I think it was poor word choice when I said “performance delta”. What I’m really getting at is, how can we measure the effect on the system, with and without HugePages?
I agree, it’s more efficient to use hugePages, overhead is reduced, etc. But is there a measurable impact? (If we were talking about improving a query’s performance, we could say, building an index consumes 1GB but saves 2 minutes every time the query is run. Is there something similar we can apply to a kernel setting like this?)
By the way, if you do write about what happens when fork() is called and hugepages is implemented, I’d love to read it. Thanks.

Comment on More on use_large_pages in Linux and 11.2.0.3 by Martin Bach

Martin Bach — Thu, 16 May 2013 10:41:57 +0000

Hi Dave,

thanks for the link to the MOS note and your comments.

Large pages don’t make applications go faster per se, they reduce overhead. Let me explain.

The immediate benefit of implementing large pages is that the kernel doesn’t need to keep track of so many small (4k) pages as compared to 2M pages. If you divide the total amount of memory by the page size you should see the benefit. Internally the kernel needs to maintain track of all pages and their state (free, dirty, …).

There are also implications about creating a copy of the page table if a process issues a fork() command. In simple terms, the bigger your SGA and the higher the number of sessions then the greater the benefit of large pages.

Let me know if you’d like to discuss further,

Martin

Comment on More on use_large_pages in Linux and 11.2.0.3 by Dave

Dave — Wed, 15 May 2013 19:57:58 +0000

Martin, great post, thank you. For the sake of completeness in your blog, MOS note 401749.1 contains a script you could use to calculate vm.nr_hugepages; note 361468.1 provides all the basics on configuration.
One question for you:
How do you determine the performance delta, once you’ve implemented HugePages? I can see all the wonderful reasons why one should move (and, in fact, we moved to HugePages a year ago, which was also a great excuse to get away from AMM). But are there any stats you know of that can tell us how smart or dumb we were in making the change?

Comment on No more excuse not to use large pages in Linux by More on use_large_pages in Linux and 11.2.0.3 « Martins Blog

More on use_large_pages in Linux and 11.2.0.3 « Martins Blog — Mon, 13 May 2013 22:45:10 +0000

[…] No more excuses not to use large pages in Linux […]

Comment on Huge Pages and Linux-Real World example by More on use_large_pages in Linux and 11.2.0.3 « Martins Blog

More on use_large_pages in Linux and 11.2.0.3 « Martins Blog — Mon, 13 May 2013 22:45:07 +0000

[…] Huge pages and real world example […]

Comment on Grid Infrastructure And Database High Availability Deep Dive Seminars 2013 by Martin Bach

Martin Bach — Thu, 09 May 2013 09:28:24 +0000

Hi Stephen,

It seems to have to do with the country selection on the website. The link didn’t return any data for me when my location was set to United States. It did fetch the detail for United Kingdom for example.