I was really sorry I missed out last year but this time I’m glad to participate and attend!

The list of great speakers includes too many to name here-you should see for yourself about who is coming to Dallas this August and why this event is unmissable.

I’m hoping to see you there!

Background and some history about processor architecture

For quite some time now the CPUs you get from AMD and Intel both are NUMA, or better: cache coherent NUMA CPUs. They all have their own “local” memory directly attached to them, in other words the memory distribution is not uniform across all CPUs. This isn’t really new, Sequent has pioneered this concept on x86 a long time ago but that’s in a different context. You really should read Scaling Oracle 8i by James Morle which has a lot of excellent content related to NUMA in it, with contributions from Kevin Closson. It doesn’t matter that it reads “8i” most of it is as relevant today as it was then.

So what is the big deal about NUMA architecture anyway? To explain NUMA and why it is important to all of us a little more background information is on order.

Some time ago processor designers and architects of industry standard hardware could no longer ignore the fact that a front side bus (FSB) proved to be a bottleneck. There were two reasons for this: it was a) too slow and b) too much data had to go over it. As one direct consequence DRAM memory has been directly attached to the CPUs. AMD has done this first with it’s Opteron processors in its AMD64 micro architecture, followed by Intel’s Nehalem micro architecture. By removing the requirement of data retrieved from DRAM to travel across a slow bus latencies could be removed.

Now imagine that every processor has a number of memory channels to which DDR3 (DDR4 could arrive soon!) SDRAM is attached to. In a dual socket system, each socket is responsible for half the memory of the system. To allow the other socket to access the corresponding other half of memory some kind of interconnect between processors is needed. Intel has opted for the Quick Path Interconnect, AMD (and IBM for p-Series) use Hyper Transport. This is (comparatively) simple when you have few sockets, up to 4 each socket can directly connect to every other without any tricks. For 8 sockets it becomes more difficult. If every socket can directly communicate with its peers the system is said to be glue-less which is beneficial. The last production glue-less system Intel released was based on the Westmere architecture. Sandy Bridge (current until approximately Q3/2013) didn’t have an eight-way glue-less variant, and this is exactly why you get Westmere-EX in the X3-8, and not Sandy Bridge as in the X3-2.

Anyway, your system will have local and remote memory. For most of us, we are not going to notice this at all since there is little point in enabling NUMA on systems with two sockets. Oracle still recommends that you only enable NUMA on 8 way systems, and this is probably the reason the oracle-validated and preinstall RPMs add “numa=off” to the kernel command line in your GRUB boot loader.

Booting with NUMA enabled

The easiest way to boot with NUMA enabled is to get to your ILOM and boot the server. As soon as the GRUB line (“booting … in x seconds”) appears, hit a key. You will be dropped into the GRUB menu. It should highlight the default boot entry (Oracle Linux Server (2.6.39.400…x86-64). Hit the “e” key to edit the directives. You should see something like this now:

root (hd0,0)
kernel /vmlinuz-2.6.39-400.xxx ....
initrd /initramfs-2.6.39-400.xxx

Move the cursor to the line starting with kernel, then hit “e” again. The cursor will move to the end of the line, where you will find the numa=off directive. Hit the backspace key to remove numa=off, then hit return (it will bring you back to the previous 3 directions), then “b” to boot this configuration.

This is useful because it doesn’t involve editing the grub menu file, and if something should break you can simply restart and are back in a known good configuration.

Now when you log in as root you will notice that NUMA is turned on!

Signs of NUMA

My lab server is an AMD 6238 dual socket workstation with 32GB of RAM. To see the effect of NUMA, you can make use of the numactl tool:

[root@ol62 ~]# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5
node 0 size: 8190 MB
node 0 free: 1637 MB
node 1 cpus: 6 7 8 9 10 11
node 1 size: 8192 MB
node 1 free: 1732 MB
node 2 cpus: 12 13 14 15 16 17
node 2 size: 8192 MB
node 2 free: 1800 MB
node 3 cpus: 18 19 20 21 22 23
node 3 size: 8176 MB
node 3 free: 1745 MB
node distances:
node   0   1   2   3
  0:  10  16  16  16
  1:  16  10  16  16
  2:  16  16  10  16
  3:  16  16  16  10

You need to know that Opteron reports twice the number of NUMA nodes than there are sockets since their 6100 series. These processors are multi-module chips on the same die. Each of the sockets has 12 cores or better: modules. AMD’s processors are somewhere between HyperThreads and cores, to which extent I can’t tell. The server reports 24 CPUs in any case.

My configuration has allocated 12295 large pages at boot time or roughly 24 GB out of 32GB available. You can see how many pages have been allocated per CPU node in the first half of the output. Luckily the memory has been requested evenly across all NUMA nodes.

The second part of the numactl output gives you the node distances in a matrix. The numbers are provided by the Operating System at boot time in form of the System Locality Table (SLIT) and cannot be changed. They indicate the cost of accessing remote memory. 10 seems to be the base value for this parameter for local access. Higher values indicate more overhead.

NUMA in SYSFS

The SYS pseudo file system is set to replace the venerable /proc file system. The SYSFS exports more information than /proc does, which is apparent when it comes to memory allocation per NUMA node. Per node NUMA statistics are in /sys/devices/system/node*

Two files are out of interest, numastat and meminfo. I won’t go into detail for numastat (yet another post will follow), but meminfo is interesting.

[root@ol62 node0]# cat meminfo
Node 0 MemTotal:        8386572 kB
Node 0 MemFree:         1685988 kB
Node 0 MemUsed:         6700584 kB
Node 0 Active:            10516 kB
Node 0 Inactive:          12704 kB
Node 0 Active(anon):       2656 kB
Node 0 Inactive(anon):        0 kB
Node 0 Active(file):       7860 kB
Node 0 Inactive(file):    12704 kB
Node 0 Unevictable:        1172 kB
Node 0 Mlocked:            1172 kB
Node 0 Dirty:                 0 kB
Node 0 Writeback:             0 kB
Node 0 FilePages:         21276 kB
Node 0 Mapped:             2960 kB
Node 0 AnonPages:          3156 kB
Node 0 Shmem:               116 kB
Node 0 KernelStack:        1384 kB
Node 0 PageTables:          528 kB
Node 0 NFS_Unstable:          0 kB
Node 0 Bounce:                0 kB
Node 0 WritebackTmp:          0 kB
Node 0 Slab:              23788 kB
Node 0 SReclaimable:       5652 kB
Node 0 SUnreclaim:        18136 kB
Node 0 AnonHugePages:         0 kB
Node 0 HugePages_Total:  3074
Node 0 HugePages_Free:   3074
Node 0 HugePages_Surp:      0

This file is similar to /proc/meminfo but only relevant for node0, i.e. the first 6 “cores” on my system. Here you can see the large page allocation on this node.

Why does this matter

When you are consolidating lots of environments to your system with lots of sockets, you should try and stick to memory locality. Keep instances on a socket if possible, today’s servers can take a lot of memory and you shouldn’t have to use remote memory this avoiding latency. I personally would use control groups to ensure my instances stay where I want them to stay. There are other ways to control memory distribution (see some of the SLOB examples) but cgroups are by far the most elegant.

Using NUMA on your system and leaving it to chance how memory is distributed will lead to difficult-to-predict performance. You might even run out of memory on a local node causing unexpected problems. As with everything, understanding and tuning a configuration is the way to go! I will run a few benchmarks next to demonstrate the difference between local and remote memory access. Unfortunately I don’t have a 4-way system available for these tests-normally you wouldn’t really worry about NUMA settings on less than four cores.

Warning

Don’t go and rush your systems to NUMA! Like I said, there is little to be gained in about 80% of all servers out there on dual-socket systems. Four-way servers might be candidates for NUMA, 8 way are candidates. By saying candidates I mean if you understand NUMA and how it can affect your application, and have really load tested it and only if it provided to be predictable, stable performance, then I would think of enabling NUMA for a production workload. There is nothing like thorough testing that can tell you how your application will perform. I guess all I want to say is that turning on NUMA can have negative performance impact as well, or even crash your Oracle instance if the memory on a NUMA node is depleted. Search MOS for NUMA to get more information.

Reference

• http://www.hardwaresecrets.com/article/324
• http://www.hardwaresecrets.com/article/19
• http://ark.intel.com – search for “products by code name”

I have previously written about the benefits and usage of large pages in Linux here:

• Huge pages and real world example
• No more excuses not to use large pages in Linux

So now as you may know there is a change to the init.ora parameter “use_large_pages” in 11.2.0.3. The parameter can take these values:

SQL> select value,isdefault
  2  from V$PARAMETER_VALID_VALUES
  3* where name = 'use_large_pages'

VALUE		     ISDEFAULT
-------------------- --------------------
TRUE		     TRUE
AUTO		     FALSE
ONLY		     FALSE
FALSE		     FALSE

There is a new value named “auto” that didn’t exist prior to 11.2.0.3. The intention is to create large pages at instance startup if possible, even if /etc/sysctl.conf doesn’t have an entry for vm.nr_hugepages at all. The risk though is that-as with dynamic creation of large pages by echoing values into /proc/sys/vm/nr_hugepages-is that you get fewer than you expect. Maybe even 0. Now I’m interested to see if that works.

So let’s have a look, my system is Oracle Linux 6.4, 64bit running virtualised. Before any database was started I checked /proc/meminfo

[root@ol64 ~]# cat /proc/meminfo
MemTotal:        8192240 kB
MemFree:         5090124 kB
Buffers:           67408 kB
Cached:          2341504 kB
SwapCached:            0 kB
Active:           816116 kB
Inactive:        2055352 kB
Active(anon):     548760 kB
Inactive(anon):   284304 kB
Active(file):     267356 kB
Inactive(file):  1771048 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        524284 kB
SwapFree:         524284 kB
Dirty:                60 kB
Writeback:             0 kB
AnonPages:        462560 kB
Mapped:           334424 kB
Shmem:            370516 kB
Slab:             103692 kB
SReclaimable:      47496 kB
SUnreclaim:        56196 kB
KernelStack:        2016 kB
PageTables:        26008 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4620404 kB
Committed_AS:    3343896 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       26480 kB
VmallocChunk:   34359700348 kB
HardwareCorrupted:     0 kB
AnonHugePages:    247808 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8128 kB
DirectMap2M:     8380416 kB

I am interested in the HugePages entries here towards the end of the list. If you have ever looked at /proc/meminfo in the previous release’s kernels (2.6.18.x to be precise) then you’ll notice it’s quite different now with a lot more information. Modern kernels are really a great step ahead. Have a look at the Outlook and References section below, this is a somewhat superficial explanation but good enough for the purpose of this article. A future post will go into more detail about SYSFS which is slated to replace the /proc file system and NUMA considerations.

Back to this article … the database I have running in my VM doesn’t use large pages, as shown in the alert.log:

Starting ORACLE instance (normal)
****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 16 MB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

RECOMMENDATION:
  Total Shared Global Region size is 2514 MB. For optimal performance,
  prior to the next instance restart increase the number
  of unused Large Pages by atleast 1257 2048 KB Large Pages (2514 MB)
  system wide to get 100% of the Shared
  Global Region allocated with Large pages
***********************************************************

So let’s change that, but dynamically and not manually. Again, a better (more predictable!) approach would be to manually add an additional 1257 large pages to /etc/sysctl.conf as recommended and reboot to ensure that they will be available when the database starts. And probably set use_large_pages to “only” to enforce their usage. But enough warnings that you probably don’t want to use the “auto” feature, I want to see this in real life!

SQL> alter system set use_large_pages=auto;
alter system set use_large_pages=auto
                 *
ERROR at line 1:
ORA-02095: specified initialization parameter cannot be modified

SQL> a  scope=spfile;
  1* alter system set use_large_pages=auto scope=spfile
SQL> /

System altered.

As you can see the parameter is static and requires an instance restart, so this is what I did next. Here is an interesting side effect of setting the parameter to “auto”: it doesn’t have an effect if you didn’t prepare the system for use of large pages in /etc/security/limits.conf. You could think that the oracle-preinstall RPM does so, but it misses the settings for “memlock”. Here is proof nothing happened:

[root@ol64 ~]# cat /proc/meminfo
MemTotal:        8192240 kB
MemFree:         5090124 kB
Buffers:           67408 kB
Cached:          2341504 kB
SwapCached:            0 kB
Active:           816116 kB
Inactive:        2055352 kB
Active(anon):     548760 kB
Inactive(anon):   284304 kB
Active(file):     267356 kB
Inactive(file):  1771048 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        524284 kB
SwapFree:         524284 kB
Dirty:                60 kB
Writeback:             0 kB
AnonPages:        462560 kB
Mapped:           334424 kB
Shmem:            370516 kB
Slab:             103692 kB
SReclaimable:      47496 kB
SUnreclaim:        56196 kB
KernelStack:        2016 kB
PageTables:        26008 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4620404 kB
Committed_AS:    3343896 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       26480 kB
VmallocChunk:   34359700348 kB
HardwareCorrupted:     0 kB
AnonHugePages:    247808 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8128 kB
DirectMap2M:     8380416 kB

HugePagesTotal is still 0, which didn’t surprise me. To allow oracle to lock memory you need to grant it the privilege. I had to edit /etc/security/limits.conf and set the memlock parameter to 5GB which is too high for my 2.5 GB SGA but setting the value a little too high doesn’t hurt at all either. The value is in kb by the way.

oracle    soft    memlock    5242880
oracle    hard    memlock    5242880

After logging out and back in as oracle I tried once more and hey-success!

Starting ORACLE instance (normal)
DISM started, OS id=11969
****************** Large Pages Information *****************
Parameter use_large_pages = AUTO

Total Shared Global Region in Large Pages = 2514 MB (100%)

Large Pages used by this instance: 1257 (2514 MB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 16 MB)
Large Pages configured system wide = 1257 (2514 MB)
Large Page size = 2048 KB
Time taken to allocate Large Pages = 0.130804 sec
***********************************************************
LICENSE_MAX_SESSION = 0

Also notice the DISM process here which is responsible for creating the large pages on the fly. This is an interesting “background process”, and Tanel Poder has mentioned it in one of his presentations already:

[root@ol64 ~]# ps -ef | grep 11969
root     11969     1  0 12:50 ?        00:00:00 ora_dism_ora11
root     12026 11911  0 12:50 pts/3    00:00:00 grep 11969
[root@ol64 bin]# ls -l $ORACLE_HOME/bin/oradism
-rwsr-x---. 1 root oinstall 71758 Sep 17  2011 oradism

It is owned by root with the setuid flag set … easy to miss when cloning a home …

Once the large pages are created, the process disappears when you start the instance a second time, and there is no mention of it in the alert.log pertaining to the startup sequence. But it has done its work.

[root@ol64 ~]# grep -i page /proc/meminfo
AnonPages:        346808 kB
PageTables:        12852 kB
AnonHugePages:    208896 kB
HugePages_Total:    1257
HugePages_Free:     1045
HugePages_Rsvd:     1045
HugePages_Surp:        0
Hugepagesize:       2048 kB

Notice that not all pages are actually in use yet, I only just started the database. Don’t worry though, 100% of the SGA are allocated in large pages as per the alert.log. Over time you will notice more and more pages being in use.

Now you can of course force the database to touch all these pages, but it’s another question whether that is a good idea. You probably don’t want to do so if you have a large SGA, the startup time can be very long. For the sake of completeness I added this here though to show you the effect in /proc/meminfo. I set pre_page_sga = true and bounced the instance:

[root@ol64 ~]# grep -i page /proc/meminfo
AnonPages:        370632 kB
PageTables:        15272 kB
AnonHugePages:    206848 kB
HugePages_Total:    1257
HugePages_Free:        3
HugePages_Rsvd:        3
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@ol64 ~]#

Now all pages are allocated straight after instance start. If you want to follow the example, I suggest you use the watch command as shown here:

watch grep -i page /proc/meminfo

Summary and a bit of warning

I personally wouldn’t rely on use_large_pages = auto in an environment I care about. It’s simply too unpredictable that you get the large pages requested, and you might fall back into 4k page mode. Planning is better than hoping-calculate the number of large pages beforehand, add them to /etc/sysctl.conf in vm.nr_hugepages and you should be almost guaranteed to have them allocated. Large pages need enough contiguous memory or otherwise the allocation may (partially) fail.

Also – large pages cannot be swapped out during memory pressure. Don’t forget you still need enough space for the PGAs and operating system! If the system starts swapping although “free” shows a lot of free memory then most likely you are using up all the 4k pages in memory.

Outlook

There is even more to be said about large pages on systems with more than 2 sockets, especially when it comes to allocating large pages per NUMA node. I’ll leave that for a future post.

Oh and yes, the large page information in /proc is only a legacy, it’s now all in SYSFS in /sys/kernel/mm/hugepages/hugepages-2048kB. Intel x86-64 supports three different page sizes: 1GB, 2048kb and 4kb.

Reference

https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt

Here is my setup. Storage is presented in 3 targets using tgtd on the host:

1. Target 1 contains 3×2 GB LUNs for OCR and voting disks in normal redundancy.
2. Target 2 contains 3×10 GB LUNs for +DATA
3. Target 2 contains 3×10 GB LUNs for +RECO

iSCSI initiators are Oracle Linux 6.4 on KVM with the host running OpenSuSE 12.3 providing the iSCSI targets. Yes, I know I’m probably the only Oracle DBA running SuSE, but to my defence I have a similar system with Oracle Linux 6.4 throughout and both work.

So besides the weird host OS there is nothing special. Since I’m lazy sometimes and don’t particularly like udev I decided to use ASMLib for device name persistence on the iSCSI LUNs. This turned out to be crucial, otherwise I’d never had written this post.

So much for the introduction

And here’s the problem. While installing Grid Infrastructure OUI allowed me to fill out all wizard interfaces and proceeded to install the binaries on all hosts. If you have installed RAC before that’s not the interesting part of the installation. It gets far more interesting when you run root.sh! Normally root.sh simply completes if you paid attention to the pre-requisites. I _have_ paid attention to them yet still the script failed on node 1! I don’t have the exact output on screen any more but the script bailed out trying to create the voting files.

Whenever something goes wrong with the installation of Grid Infrastructure you can turn to $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_$(hostname).log. The relevant section in the file contained this:

2013-04-23 14:11:12: Creating voting files
2013-04-23 14:11:12: Creating voting files in ASM diskgroup OCR
2013-04-23 14:11:12: Executing crsctl replace votedisk '+OCR'
2013-04-23 14:11:12: Executing /u01/app/11.2.0/grid/bin/crsctl replace votedisk '+OCR'
2013-04-23 14:11:12: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl replace votedisk '+OCR'
2013-04-23 14:11:12: Command output:
>  Failed to create voting files on disk group OCR.
>  Change to configuration failed, but was successfully rolled back.
>  CRS-4000: Command Replace failed, or completed with errors.
>End Command output
2013-04-23 14:11:12: Voting file add failed

Ooops. Not good-why would that fail? The ASM instance was up, the OCR has already been created. Most Clusterware commands leave a trace in $GRID_HOME/log/$(hostname)/client. I checked the last file in there but it didn’t help much:

2013-04-23 15:07:49.079: [  CRSCTL][3108575008]crsctl_format_diskgroup: diskgroup OCR creation with status 1. Please check the alert log file for ASM
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: === clsssConfigUnlock ===
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: lock(0x1cb70d0), version(1), size(1088)
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: offsets(24), activever(186647296)
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: id(14), instantiation(12), incarn(1)
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: mapoff(28), configoff(548), mapsize(512)
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: state(0), holders(0), waiters(0)
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: committimestamp(0), commitstate(0)
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: === Map (of first 7 entries) ===
2013-04-23 15:07:49.082: [ CSSCLNT][3108575008]clsssConfigLockTrace: 000 000 000 000 000 000 000

Except for line 1-that is a very obvious pointer. Maybe there is something in the ASM instance’s alert.log?

2013-04-23 15:07:48.733000 +01:00
NOTE: [crsctl.bin@rac11gr2node1.example.com (TNS V1-V3) 13092] opening OCR file
NOTE: updated gpnp profile ASM diskstring:
NOTE: Creating voting files in diskgroup OCR
NOTE: Voting File refresh pending for group 1/0x20cf9032 (OCR)
NOTE: Attempting voting file creation in diskgroup OCR
ERROR: Could not create voting files. It spans across 161 AUs (max supported is 64 AUs)
ERROR: Voting file allocation failed for group OCR
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13100.trc:
ORA-15303: Voting files could not be created in diskgroup OCR due to small Allocation Unit size
2013-04-23 15:07:51.278000 +01:00
NOTE: Attempting voting file refresh on diskgroup OCR

That was it! But what’s so special about the disk group? Something started to dawn on me…I have recently spent quite some time on 4k sector size disks and their implications. And my hard disks have recently been replaced … here’s what I got from ASM:

SQL> select name,sector_size from v$asm_diskgroup;

NAME                           SECTOR_SIZE
------------------------------ -----------
OCR                                   4096

So sure enough, my disk group uses 4k sectors, even though at no point I asked it to do so. In fact up to today I struggled to create a disk group with a 4k sector size for lack of supporting hardware! So this is the first time I see those, my hard disks in the lab server must be pretty new then. There are many ways to check for the block size of your LUN, this time I chose fdisk (this only works for LUNs that use the MBR format, if the LUN has been initialised with a GPT you need to install parted instead)

[root@rac11gr2node1 client]# fdisk -lu /dev/sda

Disk /dev/sda: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1          531712     4191385     1829837   83  Linux
[root@rac11gr2node1 client]#

Notice this line:

Sector size (logical/physical): 512 bytes / 4096 bytes

And there exactly lies the problem. ASMLib in my version cannot deal with a 4k sector size properly it seems. The problem is known, just search for ORA-15303 in My Oracle Support. It seems particular to LUNs/disks with different logical and physical block sizes though-refer to the MOS note for more information. For your reference, here are my versions:

[oracle@rac11gr2node1 ~]$ rpm -qa| grep oracleasm
oracleasmlib-2.0.4-1.el6.x86_64
oracleasm-support-2.1.8-1.el6.x86_64
[oracle@rac11gr2node1 ~]$

The strange thing is that the ASM instance with the single disk group has been created successfully, as shown by the log (note the disk list in line 1):

2013-04-23 14:10:54: Executing as oracle: /u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList 'ORCL:OCR1,ORCL:OCR2,ORCL:O
CR3' -redundancy NORMAL -configureLocalASM -au_size 1
2013-04-23 14:10:54: Running as user oracle: /u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList 'ORCL:OCR1,ORCL:OCR2,ORCL:OCR3' -redundancy NORMAL -configureLocalASM -au_size 1
2013-04-23 14:10:54:   Invoking "/u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList 'ORCL:OCR1,ORCL:OCR2,ORCL:OCR3' -redundancy NORMAL -configureLocalASM -au_size 1" as user "oracle"
2013-04-23 14:10:54: Executing /bin/su oracle -c "/u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList 'ORCL:OCR1,ORCL:OCR2,ORCL:OCR3' -redundancy NORMAL -configureLocalASM -au_size 1"
2013-04-23 14:10:54: Executing cmd: /bin/su oracle -c "/u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList 'ORCL:OCR1,ORCL:OCR2,ORCL:OCR3' -redundancy NORMAL -configureLocalASM -au_size 1"
2013-04-23 14:11:09: Command output:
>
>  ASM created and started successfully.
>
>  Disk Group OCR created successfully.
>
>End Command output

After spending a little while thinking what to do I decided to bite the bullet and start from scratch. The deinstall scripts again worked really well in my case.

Instead of ASMLib I used udev for the second attempt. The trick with udev is to find something to map. The scsi_id command for example is a great help in determining disk attributes, but there are others too. In my case, all I wanted to achieve is to change permissions of the iSCSI disks to oracle:dba and 0660. In my lab environment I didn't use multiple paths to the iSCSI target, and I didn't care about symlinks. You get it, this isn't 100% realistic ... Note that In KVM disks are named vd*, like /dev/vda for the first one etc. if you are using virtio, the para-virtualised drivers. The iSCSI disks ended up being called /dev/sd* which makes it easy to define an asm_diskstring.
[root@rac11gr2node1 sys]# /sbin/scsi_id --whitelisted --replace-whitespace --page=0x80 --device=/dev/sdg --export
ID_SCSI=1
ID_VENDOR=IET
ID_VENDOR_ENC=IET\x20\x20\x20\x20\x20
ID_MODEL=VIRTUAL-DISK
ID_MODEL_ENC=VIRTUAL-DISK
ID_REVISION=0001
ID_TYPE=disk
ID_SERIAL_RAW="SIET     VIRTUAL-DISK"
ID_SERIAL=SIET_VIRTUAL-DISK
ID_SERIAL_SHORT=

So using the vendor string and the model attribute from SYSFS will most likely work. Please note that this is a rather simplistic model and wouldn't necessarily work with Fibre Channel attached disks or multipathing. The resulting udev rule in /etc/udev/rules/99-asm.rules was:
KERNEL=="sd[a-z]*", BUS=="scsi", SYSFS{vendor}=="IET", SYSFS{model}=="VIRTUAL-DISK", OWNER="oracle", GROUP="dba" MODE="0660"

I also deconfigured ASMLib on all nodes just to be sure:
[root@rac11gr2node1 u01]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]:
Default group to own the driver interface [dba]:
Start Oracle ASM library driver on boot (y/n) [y]: n
Scan for Oracle ASM disks on boot (y/n) [y]: n
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@rac11gr2node1 u01]#

Running udevadm trigger loaded the new rules and indeed, the /dev/sd*1 devices now were owned by oracle:dba.
Then I ran through OUI again and wanted to assign the previously used disks again, but they didn't appear as candidates :( OK so the deconfig didn't zero out the disk headers. Not a problem, oracleasm has a "deletedisk" command that can be used for this exact purpose. You just need to be really sure it's the correct disk you are zeroing out, otherwise someone else will surely complain (you have been warned!).
The rest was simple. The ASM instance has again been created successfully, note the difference in disk names this time (my asm_diskstring was set to /dev/sd*1):
2013-04-23 16:29:29: Executing as oracle: /u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList '/dev/sda1,/dev/sdb1,/dev/sdc1' -redundancy NORMAL -diskString '/dev/sd*1' -configureLocalASM -au_size 1
2013-04-23 16:29:29: Running as user oracle: /u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList '/dev/sda1,/dev/sdb1,/dev/sdc1' -redundancy NORMAL -diskString '/dev/sd*1' -configureLocalASM -au_size 1
2013-04-23 16:29:29:   Invoking "/u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList '/dev/sda1,/dev/sdb1,/dev/sdc1' -redundancy NORMAL -diskString '/dev/sd*1' -configureLocalASM -au_size 1" as user "oracle"
2013-04-23 16:29:29: Executing /bin/su oracle -c "/u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList '/dev/sda1,/dev/sdb1,/dev/sdc1' -redundancy NORMAL -diskString '/dev/sd*1' -configureLocalASM -au_size 1"
2013-04-23 16:29:29: Executing cmd: /bin/su oracle -c "/u01/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList '/dev/sda1,/dev/sdb1,/dev/sdc1' -redundancy NORMAL -diskString '/dev/sd*1' -configureLocalASM -au_size 1"

And this time the voting disks were created as well:
CRS-2676: Start of 'ora.diskmon' on 'rac11gr2node2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac11gr2node2' succeeded

ASM created and started successfully.

Disk Group OCR created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 9d9a9577a66a4f6fbf267b125dc7f4a3.
Successful addition of voting disk 26e5ff2544084ff0bfb9e753b090ec22.
Successful addition of voting disk 38a820cd4b424f4ebf514370168fa499.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   9d9a9577a66a4f6fbf267b125dc7f4a3 (/dev/sda1) [OCR]
 2. ONLINE   26e5ff2544084ff0bfb9e753b090ec22 (/dev/sdb1) [OCR]
 3. ONLINE   38a820cd4b424f4ebf514370168fa499 (/dev/sdc1) [OCR]
Located 3 voting disk(s).

Very nice-job done. You should refer back to the relevant MOS note to read more about the problem but for now be advised that 4k sector sizes can cause surprises.

   ]]>
			http://martincarstenbach.wordpress.com/2013/04/29/4k-sector-size-and-grid-infrastructure-11-2-installation-gotcha/feed/
		3
	
		
			Martin
		
	
		
		
		http://martincarstenbach.wordpress.com/2013/04/24/grid-infrastructure-and-database-high-availability-deep-dive-seminars-2013/
		http://martincarstenbach.wordpress.com/2013/04/24/grid-infrastructure-and-database-high-availability-deep-dive-seminars-2013/#comments
		Wed, 24 Apr 2013 12:40:31 +0000
		Martin Bach
				
		
		

		http://martincarstenbach.wordpress.com/?p=1564
		]]>
				So this is a little bit of a plug for myself and Enkitec but I’m running my Grid Infrastructure And Database High Availability Deep Dive Seminars again for Oracle University. This time these events are online, so no need to come to a classroom at all.

Here is the short description of the course:
Providing a highly available database architecture fit for today’s fast changing requirements can be a complex task. Many technologies are available to provide resilience, each with its own advantages and possible disadvantages. This seminar begins with an overview of available HA technologies (hard and soft partitioning of servers, cold failover clusters, RAC and RAC One Node) and complementary tools and techniques to provide recovery from site failure (Data Guard or storage replication).
In the second part of the seminar, we look at Grid Infrastructure in great detail. Oracle Grid Infrastructure is the latest incarnation of the Clusterware HA framework which successfully powers every single 10g and 11g RAC installation. Despite its widespread implementation, many of its features are still not well understood by its users. We focus on Grid Infrastructure, what it is, what it does and how it can be put to best use, including the creation of an active/passive cold failover cluster for web and database resources.
If you are interested I would like to invite you to head over to the Oracle University website here which has a more extensive synopsis and all the detail you need:
http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getCourseDesc?dc=D81641_2043034
UPDATE: I received several emails and comments that the above link does not work. I couldn’t reproduce this until today. It appears to be an issue with the country selection. If you have USA selected in the top right corner the link won’t work, switching to United Kingdom (my preference) will fetch the course detail. I don’t quite understand as to why that is the case since the class is virtual and not depending on a country…
I hope to hear from you during the course!

   ]]>
			http://martincarstenbach.wordpress.com/2013/04/24/grid-infrastructure-and-database-high-availability-deep-dive-seminars-2013/feed/
		7
	
		
			Martin
		
	
		
		
		http://martincarstenbach.wordpress.com/2013/04/23/limiting-the-degree-of-parallelism-via-resource-manager-and-a-gotcha/
		http://martincarstenbach.wordpress.com/2013/04/23/limiting-the-degree-of-parallelism-via-resource-manager-and-a-gotcha/#comments
		Tue, 23 Apr 2013 10:54:36 +0000
		Martin Bach
				
		
		
		
		
		
		

		http://martincarstenbach.wordpress.com/?p=1535
		]]>
				This might be something very obvious for the reader but I had an interesting revelation recently when implementing parallel_degree_limit_p1 in a resource consumer group. My aim was to prevent users mapped to a resource consumer group from executing any query in parallel. The environment is fictional, but let’s assume that it is possible that maintenance operations for example leave indexes and tables decorated with a parallel x attribute. Another common case is the restriction of PQ resource to users to prevent them from using all the machine’s resources.

This can happen when you perform an index rebuild for example in parallel to speed the operation up. However the DOP will stay the same with the index after the maintenance operation, and you have to explicitly set it back:
SQL> alter index I_T1$ID1 rebuild parallel 4;

Index altered.

SQL> select index_name,degree from ind where index_name = 'I_T1$ID1';

INDEX_NAME		       DEGREE
------------------------------ ----------------------------------------
I_T1$ID1		       4

SQL>

Now when you select from the table and the index is involved you can end up with a parallel query (PQ). Admittedly this example is a little contrived but not too far from reality either.
SQL> select count(1) from t1 a, t1 b
 2 where a.object_id = b.object_id;

  COUNT(1)
----------
     74511

Now I’m getting the execution plan for this statement-I expect it to be parallel:
SQL> select * from table(dbms_xplan.display_cursor);

SQL_ID	66xstr294k2f6, child number 0
-------------------------------------
select count(1) from t1 a, t1 b where a.object_id = b.object_id

Plan hash value: 3484396843

----------------------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name     | Rows	| Bytes | Cost (%CPU)| Time	|    TQ  |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT	     |		|	|	|   127 (100)|		|	 |	|	     |
|   1 |  SORT AGGREGATE 	     |		|     1 |    26 |	     |		|	 |	|	     |
|   2 |   PX COORDINATOR	     |		|	|	|	     |		|	 |	|	     |
|   3 |    PX SEND QC (RANDOM)	     | :TQ10001 |     1 |    26 |	     |		|  Q1,01 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE	     |		|     1 |    26 |	     |		|  Q1,01 | PCWP |	     |
|*  5 |      HASH JOIN		     |		|   134K|  3414K|   127   (4)| 00:00:02 |  Q1,01 | PCWP |	     |
|   6 |       PX BLOCK ITERATOR      |		|   134K|  1707K|    61   (0)| 00:00:01 |  Q1,01 | PCWC |	     |
|*  7 |        INDEX FAST FULL SCAN  | I_T1$ID1 |   134K|  1707K|    61   (0)| 00:00:01 |  Q1,01 | PCWP |	     |
|   8 |       PX RECEIVE	     |		|   134K|  1707K|    61   (0)| 00:00:01 |  Q1,01 | PCWP |	     |
|   9 |        PX SEND BROADCAST     | :TQ10000 |   134K|  1707K|    61   (0)| 00:00:01 |  Q1,00 | P->P | BROADCAST  |
|  10 | 	PX BLOCK ITERATOR    |		|   134K|  1707K|    61   (0)| 00:00:01 |  Q1,00 | PCWC |	     |
|* 11 | 	 INDEX FAST FULL SCAN| I_T1$ID1 |   134K|  1707K|    61   (0)| 00:00:01 |  Q1,00 | PCWP |	     |
----------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   5 - access("A"."OBJECT_ID"="B"."OBJECT_ID")
   7 - access(:Z>=:Z AND :Z=:Z AND :Z<=:Z)
OK, that’s been done in parallel. But I don’ want it to run in parallel!
So the task at hand is to explicitly disable PQ for users connecting via service OLTP_SRVC. I spare you the detail of the creation of the consumer group, for this example I have created the group and assigned user martin to it. The mapping to the consumer group is based on the service name. I had to explicitly grant my user the right to switch consumer group as shown here:
BEGIN
 dbms_resource_manager_privs.grant_switch_consumer_group(
  GRANTEE_NAME   => 'MARTIN',
  CONSUMER_GROUP => 'OLTP_GRP',
  GRANT_OPTION   => FALSE);
END;
/

Then it’s time to create the plan and corresponding directives. The relevant piece of code is shown here:
BEGIN
 dbms_resource_manager.clear_pending_area();
 dbms_resource_manager.create_pending_area();

 -- more code

 dbms_resource_manager.create_plan_directive(
 plan => 'someplan',
 group_or_subplan => 'OLTP_GRP',
 mgmt_p2 => 65,
 comment => 'no parallel!',
 parallel_degree_limit_p1 => 0
 );

 -- more code

 dbms_resource_manager.validate_pending_area();
 dbms_resource_manager.submit_pending_area();
end;
/

After the familiar “PL/SQL procedure completed successfully” message I enabled someplan, I was all ready to enjoy the success-but wait!
There’s something strange, at least to me. I connected using the service and executed the query again but NO CHANGE. The optimizer still uses a parallel execution plan:
SQL> select * from table(dbms_xplan.display_cursor);
SQL_ID	66xstr294k2f6, child number 0
-------------------------------------
select count(1) from t1 a, t1 b where a.object_id = b.object_id

Plan hash value: 3484396843

----------------------------------------------------------------------------------------------------------------------
| Id  | Operation		     | Name	| Rows	| Bytes | Cost (%CPU)| Time	|    TQ  |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT	     |		|	|	|   127 (100)|		|	 |	|	     |
|   1 |  SORT AGGREGATE 	     |		|     1 |    26 |	     |		|	 |	|	     |
|   2 |   PX COORDINATOR	     |		|	|	|	     |		|	 |	|	     |
|   3 |    PX SEND QC (RANDOM)	     | :TQ10001 |     1 |    26 |	     |		|  Q1,01 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE	     |		|     1 |    26 |	     |		|  Q1,01 | PCWP |	     |
|*  5 |      HASH JOIN		     |		|   134K|  3414K|   127   (4)| 00:00:02 |  Q1,01 | PCWP |	     |
|   6 |       PX BLOCK ITERATOR      |		|   134K|  1707K|    61   (0)| 00:00:01 |  Q1,01 | PCWC |	     |
|*  7 |        INDEX FAST FULL SCAN  | I_T1$ID1 |   134K|  1707K|    61   (0)| 00:00:01 |  Q1,01 | PCWP |	     |
|   8 |       PX RECEIVE	     |		|   134K|  1707K|    61   (0)| 00:00:01 |  Q1,01 | PCWP |	     |
|   9 |        PX SEND BROADCAST     | :TQ10000 |   134K|  1707K|    61   (0)| 00:00:01 |  Q1,00 | P->P | BROADCAST  |
|  10 | 	PX BLOCK ITERATOR    |		|   134K|  1707K|    61   (0)| 00:00:01 |  Q1,00 | PCWC |	     |
|* 11 | 	 INDEX FAST FULL SCAN| I_T1$ID1 |   134K|  1707K|    61   (0)| 00:00:01 |  Q1,00 | PCWP |	     |
----------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   5 - access("A"."OBJECT_ID"="B"."OBJECT_ID")
   7 - access(:Z>=:Z AND :Z=:Z AND :Z<=:Z)
I made sure the consumer group was indeed OLTP_GRP (visible from v$session). I flushed the statement from the shared pool (as described here for example), when that didn’t help I flushed the flushed the whole shared pool (in my lab-don’t just flush the shared pool!) but still no change. I proverbially scratched my head a for moment and then decided to check if there wasn’t any more information about DBRM and its decisions available. Sure enough there is (the Oracle DBMS is really well instrumented)
So whenever you are in doubt you can check the meta information about resource manager: v$resource_consumer_group:
SQL> select name,pqs_completed,pq_servers_used from V$RSRC_CONSUMER_GROUP
  2  where name = 'OLTP_GRP'
  3  /

NAME                             PQS_COMPLETED PQ_SERVERS_USED
-------------------------------- ------------- ---------------
OLT_GRP                                      1               0

SQL>

Aha! So it appears that there is a parallel execution plan even though it doesn’t execute in parallel. I would have expected a “parallel force serial” in the plan but there is no such thing. Also, when using Real Time SQL Monitor you won’t see any information about a DOP (requested/used) so there is additional proof that the execution was performed in serial. What initially left me puzzled though is that there was no serial execution plan.
Conclusion
The observed behaviour makes complete sense once you think about it for a moment. The optimizer does not know about the DBRM restriction, and it won’t kick in until run-time. When the optimizer hard-parses a statement it believes it has all the required resources available in form of the parallel% parameters, but that is not the case. I assume that their implementation allowed the Oracle engineers to use a more generic code to implement DOP restrictions, out of which DOP=0 really only is an edge case. I will test a little more if there is an overhead with the approach. In the meantime, if you really need this you could use the following login trigger for example (if you have no control over the code of course!)
SQL> get trigger
  1  create or replace trigger force_serial
  2  after logon on database
  3  declare
  4   v_service_name varchar2(100);
  5  begin
  6   v_service_name := sys_context('userenv','service_name');
  7
  8   if v_service_name = 'OLTP_SRV' then
  9    execute immediate 'alter session disable parallel query';
 10
 11   end if;
 12
 13* end;
SQL>

This is an example only and must complement the DBRM settings shown above. Admittedly it is not the most elegant way of forcing a truly serial execution and it doesn’t prevent a user from overriding the settings in his session but if your users cannot execute “alter session” than it helps a little.

   ]]>
			http://martincarstenbach.wordpress.com/2013/04/23/limiting-the-degree-of-parallelism-via-resource-manager-and-a-gotcha/feed/
		4
	
		
			Martin
		
	
		
		
		http://martincarstenbach.wordpress.com/2013/04/21/ougn-spring-seminar-2013/
		http://martincarstenbach.wordpress.com/2013/04/21/ougn-spring-seminar-2013/#comments
		Sun, 21 Apr 2013 18:38:36 +0000
		Martin Bach
				

		http://martincarstenbach.wordpress.com/?p=1542
		]]>
				The annual conference held by the Oracle User Group in Norway has once again been just great. It was the second year I went and I have to admit that it was every bit as good as last year, and that’s holding a very high standard.

The combination of such great hosts, great speakers and a wonderful atmosphere make this one of the best conferences to attend in Europe. The added benefit of being on a boat makes it a great opportunity to meet the speakers and to hang out with during dinner and after the sessions. Unfortunately I had to leave a day early and write these lines while on a train back home.
So how has it been? Unlike last year I flew in on the day of the event on Wednesday morning into Oslo together with Martin Widlake. Martin Nash and I successfully pitched RAC Attack II to the event organizers and Martin started off in his usual stride (I am referring to Martin Nash here when mentioning “Martin” by the way). There were a few attendees less this year compared to the first RAC Attack we did but Martin has prepared cool stuff to show those who came to the sessions.
We had support from Frits Hoogland and Bjorn Rost at various parts of the event which lead to a lot of interesting discussions around workload management in RAC. While we were overcoming a network particularity with the hotel’s wireless network I used a parts of my “Grid Infrastructure and High Availability Deep Dive” seminar to demonstrate the message we wanted to put across: workload management is _the_ way to consolidate applications on RAC. It is also the only way to allow for rolling patches of infrastructure and applications. What good is RAC if you are experiencing downtime in your application because of restarts of the application servers? Sadly I have not come across an application yet that was truly RAC-aware. Often though you see FUD take over and the benefits of RAC cannot be used.
Martin went into a lot of detail in his preparations to demonstrate the usefulness of using Oracle’s Universal Connection Pool (UCP) as well as .Net for Fast Connection Failover (FCF). In my very personal opinion anyone seriously considering RAC for availability definitely has to spend time on FCF, or otherwise save the license and use an active/passive cluster or RAC One Node instead. With the increasing importance of Exadata and the Oracle Database Appliance RAC awareness is really needed, especially with developers. In my opinion it is not enough to simply use Transparent Application Failover and stop there-TAF is so 1990s! When using TAF then only as a stopgap measure until the application has been properly ported to RAC.
I would like to thank the attendees for their support and the great time I had. It was particularly cool to put faces to twitter handles, like in the case of Philippe Fierens who sat in the lab.
There are a few things to take away and discuss with the rest of the RAC Attack organisers such as the move away from VMWare Server which doesn’t really work with Windows > XP. Other than that I still believe in the format and want to thank Jeremy Schneider who initiated it.
Thursday we boarded the boat to start the cruise to Kiel which gives you plenty of time to chat with people. The conference started great with many interesting sessions to attend. I would like to give credit to Joel Goodman, who presented in quite rough waters as if it were plain sailing. His subject was Resource Management and RAC and I have not yet had such a precise explanation of global enqueue and buffer cache handling in RAC before. Should he repeat this talk at another user group you should definitely go as it is so worth it.
There were other great sessions Martin Nash already covered in his summary so if you are interested in knowing what you missed, go over to his blog post now.
Friday morning it was time for my talk about emerging trends in storage and from my personal point of view I was happy with it. Of course the feedback forms will tell me if I’m right with my gut feeling. I then went to see Doug Burns with his talk about 10053 trace files and really liked it. But then I like the way Doug presents anyway, and he is one of these speakers you want to watch out for on any conference agenda. After this I had to leave the boat since we had arrived in Kiel in the meantime.
The only regret I have was not being able to make it for the dinner on Thursday evening but my body felt in no condition to consume food or drinks. I wish I could have talked to friends more than I managed to.
Anyway, if my papers next time are accepted I’m sure to be back. Thanks for everyone at OUGN for making this such a wonderful experience.

   ]]>
			http://martincarstenbach.wordpress.com/2013/04/21/ougn-spring-seminar-2013/feed/
		0
	
		
			Martin
		
	
		
		
		http://martincarstenbach.wordpress.com/2013/03/20/signed-on-the-dotted-line-enkitec/
		http://martincarstenbach.wordpress.com/2013/03/20/signed-on-the-dotted-line-enkitec/#comments
		Wed, 20 Mar 2013 20:54:56 +0000
		Martin Bach
				

		http://martincarstenbach.wordpress.com/?p=1512
		]]>
				I had to think of ‏@OyvindIsene, a great ambassador of the Norwegian Oracle User Group when I typed the heading for this post. Unlike him I have not actively been looking for new challenges but sometimes things just develop, and in my case that was a great turn of events. I am very happy to have signed on the dotted line and in a couple of weeks will join Enkitec in Europe.

How did that happen? During an Oracle conference I met Andy Colvin together with some of his colleagues during a break in the busy schedule. I already knew and respected Enkitec as a great company with lots of seriously experienced DBAs. I feel fortunate to actually know some of them already from email and other social media exchanges.
Andy and I have exchanged a few tweets in the past and I really like his blog so I was curious to meet him in person. I haven’t yet had the opportunity to go and speak at an American conference so anytime someone I know from the other side of the Atlantic comes to Europe I try to meet up. I had a great time but unfortunately had to run since my talk started a few minutes later. It was quite funny actually although I’m not so sure if my presentation was up to my own expectations. The conversations I had made a lasting impression on me.
Over the cause of the next months we remained in contact, and I had the great pleasure to meet Kerry Osborne together with Andy a little later and that was when I seriously thought that I wanted to join a team I do admire. Now with all the paperwork done and dusted, and having signed I can’t wait to get started.

   ]]>
			http://martincarstenbach.wordpress.com/2013/03/20/signed-on-the-dotted-line-enkitec/feed/
		6
	
		
			Martin
		
	
		
		
		http://martincarstenbach.wordpress.com/2013/01/03/oracle-linux-support-in-esxi/
		http://martincarstenbach.wordpress.com/2013/01/03/oracle-linux-support-in-esxi/#comments
		Thu, 03 Jan 2013 22:09:51 +0000
		Martin Bach
				
		

		http://martincarstenbach.wordpress.com/?p=1500
		]]>
				For quite some time now I am using ESXi 5 update 1 for my lab server and I’m very happy with it. In my lab environment I am not too picky what to run and do not worry about support too much. It’s not production!

One area of concern has been the support for Oracle’s own kernel: UEK or Unbreakable Enterprise Kernel. UEK comes in two editions, one based on 2.6.32, just like Red Hat’s kernel for Red Hat 6. The difference is that you can get UEK/1 (2.6.32.xxx) for Oracle Linux 5.x as well instead of 2.6.18xxx which is otherwise the default.
Oracle’s second iteration of kernel UEK is unsurprisingly named UEK2 and it’s initially based on 3.x but keeps the name to 2.6.39.x for compatibility reasons. UEK2 has some really nice features taken from the Upstream kernel and it is also supported for the Oracle database.
Until not too long ago UEK was not supported by VMware ESXi, but this has changed without me taking notice at first. Thanks to a tweet by @dba_emc2 (Allan Robertson) I learned more about the change in the support policy. One interesting blog post from VMware is found here:
http://blogs.vmware.com/guestosguide/2012/09/unbreakable-enterprise-kernel-for-oracle-linux.html
This post only mentions UEK, but does not clearly state whether UEK or UEK2 or both will be supported. The VMware Compatibility Guide has more information at http://www.vmware.com/resources/compatibility/search.php

In the search, enter “unbreakable” to be directed the the relevant certification information
It turns out it is (at the time of this writing) UEK 2 actually which is great news for me!
Supported versions of ESXi are 5 u1, 5u2 and 5.1 at the time of writing
Support date is listed as 09/2012
There are even specific installation instruction but they don’t go over and above what you would normally do
What’s very interesting is that the paravirtualised drivers for SCSI (vSCSI) and VMXNet3 are supported too
You can also add virtual CPUs and memory while the VM is up (with the proper VM settings, I think hot-adding these is deactivated by default)

Enjoy! I will try to install Oracle Linux 6.3 – which is the first to my knowledge that boots UEK2 by default – next and install the VMware tools. Let’s see how that goes.

   ]]>
			http://martincarstenbach.wordpress.com/2013/01/03/oracle-linux-support-in-esxi/feed/
		3
	
		
			Martin
		
	
		
		
		http://martincarstenbach.wordpress.com/2012/12/06/adding-disks-to-vmware-workstation-8-on-the-fly-on-rhel-6/
		http://martincarstenbach.wordpress.com/2012/12/06/adding-disks-to-vmware-workstation-8-on-the-fly-on-rhel-6/#comments
		Thu, 06 Dec 2012 21:03:50 +0000
		Martin Bach
				
		

		http://martincarstenbach.wordpress.com/?p=1476
		]]>
				Although this post is primarily written for users of VMware Workstation 8 it is applicable for any RedHat 6 clone and adding disks with single path on the fly. Multipathing requires additional setup in dm-multipath or the vendor multipathing software which I won’t cover here. A quick hint though: you need to set disk.EnableUUID = “TRUE” in your VM’s config file for scsi_id to return a value.

The situation is common: you created a virtual machine and need more storage. Hopefully you created it using LVM which would allow you to add the new disk to an existing volume group followed by a resize operation of the logical volume which is short on space. But before you can do this you have to add a new LUN to your setup-here is how you can do this without rebooting the VM.
First I recommend you install lsscsi (for convenience, not really necessary) and the sg3_utils:
[root@server1 ~]# yum install lsscsi.x86_64 sg3_utils.x86_64
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package lsscsi.x86_64 0:0.23-2.el6 will be installed
---> Package sg3_utils.x86_64 0:1.28-4.el6 will be installed
--> Processing Dependency: sg3_utils-libs = 1.28-4.el6 for package: sg3_utils-1.28-4.el6.x86_64
--> Processing Dependency: libsgutils2.so.2()(64bit) for package: sg3_utils-1.28-4.el6.x86_64
--> Running transaction check
---> Package sg3_utils-libs.x86_64 0:1.28-4.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                                      Arch                                 Version                                    Repository                           Size
========================================================================================================================================================================
Installing:
 lsscsi                                       x86_64                               0.23-2.el6                                 local                                38 k
 sg3_utils                                    x86_64                               1.28-4.el6                                 local                               470 k
Installing for dependencies:
 sg3_utils-libs                               x86_64                               1.28-4.el6                                 local                                51 k

Transaction Summary
========================================================================================================================================================================
Install       3 Package(s)

Total download size: 559 k
Installed size: 1.4 M
Is this ok [y/N]: y
Downloading Packages:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                    25 MB/s | 559 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : sg3_utils-libs-1.28-4.el6.x86_64                                                                                                                     1/3
  Installing : sg3_utils-1.28-4.el6.x86_64                                                                                                                          2/3
  Installing : lsscsi-0.23-2.el6.x86_64                                                                                                                             3/3

Installed:
  lsscsi.x86_64 0:0.23-2.el6                                                        sg3_utils.x86_64 0:1.28-4.el6

Dependency Installed:
  sg3_utils-libs.x86_64 0:1.28-4.el6

Complete!

With this done it is easy to check the attached SCSI devices, and on my system I found these:
[root@server1 ~]# lsscsi
[0:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda
[0:0:1:0] disk VMware, VMware Virtual S 1.0 /dev/sdb
[2:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0

This means there are two virtual disks and a virtual CD-ROM. Now it’s time to add the new disk. Do so using the user VMware workstation interface. Once completed, you need to rescan the scsi bus:
[root@server1 ~]# rescan-scsi-bus.sh
Host adapter 0 (mptspi) found.
Host adapter 1 (ata_piix) found.
Host adapter 2 (ata_piix) found.
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: VMware,  Model: VMware Virtual S Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 02
Scanning for device 0 0 1 0 ...
OLD: Host: scsi0 Channel: 00 Id: 01 Lun: 00
      Vendor: VMware,  Model: VMware Virtual S Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 02
Scanning for device 0 0 2 0 ...
NEW: Host: scsi0 Channel: 00 Id: 02 Lun: 00
      Vendor: VMware,  Model: VMware Virtual S Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 02
Scanning for device 0 0 2 0 ...
OLD: Host: scsi0 Channel: 00 Id: 02 Lun: 00
      Vendor: VMware,  Model: VMware Virtual S Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 02
Scanning host 1 channels  0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 channels  0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning for device 2 0 0 0 ...
OLD: Host: scsi2 Channel: 00 Id: 00 Lun: 00
      Vendor: NECVMWar Model: VMware IDE CDR10 Rev: 1.00
      Type:   CD-ROM                           ANSI SCSI revision: 05
0 new device(s) found.
0 device(s) removed.

A new disk has been found, as shown by lscsi:
[root@server1 ~]# lsscsi
[0:0:0:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sda
[0:0:1:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdb
[0:0:2:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdc
[2:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[root@server1 ~]#

Very nice-from there on it’s just another disk you can format using parted or fdisk. And you are done.

   ]]>
			http://martincarstenbach.wordpress.com/2012/12/06/adding-disks-to-vmware-workstation-8-on-the-fly-on-rhel-6/feed/
		5
	
		
			Martin