12.2 New Feature: the FLEX ASM disk group part 4

Flex Disk Group Properties

In the previous 3 parts I shared my investigation into ASM Flex Disk Groups, Quota Groups, File Groups, and how Quota Groups actually enforce space limits. What I haven’t discussed yet was changing properties of a File Group and the effects thereof. Properties I have in mind are related to the protection level, as discussed in the official documentation-Automatic Storage Management Administrator’s Guide, Administering Oracle ASM Disk Groups. There are of course other properties as well (and you’ll find a link to all of the modifiable properties later in this post), but they are out of scope for this investigation.

A disk group with Flex Redundancy can be set up with all protection levels (3-way mirror, 2-way mirror, unprotected), and by default uses 2-way mirroring. Unlike other types of disk groups, you can change the protection levels for individual (pluggable) databases within the Flex Disk group. This is best shown with an example. Continuing the story from my previous blog posts, here’s the setup again for your convenience.

SQL> select filegroup_number,name,guid from v$asm_filegroup

FILEGROUP_NUMBER NAME                 GUID
---------------- -------------------- --------------------------------
               0 DEFAULT_FILEGROUP
               1 CDB_CDB$ROOT         4700A987085A3DFAE05387E5E50A8C7B
               2 CDB_PDB$SEED         536DF51E8E28221BE0534764A8C0FD81
               3 PDB1                 537B677EF8DA0F1AE0534764A8C05729
               4 ORCL_CDB$ROOT        4700A987085A3DFAE05387E5E50A8C7B
               5 ORCL_PDB$SEED        537E63B952183748E0534764A8C09A7F
               6 PDB1_0001            537EB5B87E62586EE0534764A8C05530

7 rows selected. 

The above listing shows all my File Groups in my ASM instance. This post is about changing attributes of filegroup number 6, PDB1_0001. It will be important to understand which files pertain to the filegroup later; here’s the list:

SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
  2  from v$asm_file where filegroup_number = 6;
  
FILE_NUMBER  BYTES      SPACE      TYPE      REDUNDANCY  REDUNDANCY_LOWERED  STRIPED  REMIRROR
309          104865792  218103808  DATAFILE  MIRROR      U                   COARSE   N
310          262152192  541065216  DATAFILE  MIRROR      U                   COARSE   N
311          419438592  859832320  DATAFILE  MIRROR      U                   COARSE   N
312          67117056   142606336  TEMPFILE  MIRROR      U                   COARSE   N

The documentation is correct: the default redundancy for the datafiles and tempfile is “mirror”.

Filegroup properties

The ability to change redundancy and other properties within the disk group hinges on the fact that you have File Groups. Properties that belong to a File Group can be listed either via the SQL interface, or asmcmd. The latter is shown first:

ASMCMD> lsfg -G flex --filegroup PDB1_0001
File Group  Disk Group  Property    Value   File Type                  
PDB1_0001   FLEX        PRIORITY    MEDIUM                             
PDB1_0001   FLEX        STRIPING    COARSE  CONTAINER                  
PDB1_0001   FLEX        STRIPING    FINE    CONTROLFILE                
PDB1_0001   FLEX        REDUNDANCY  MIRROR  DATAFILE                   
PDB1_0001   FLEX        STRIPING    COARSE  DATAFILE                   
PDB1_0001   FLEX        REDUNDANCY  MIRROR  ONLINELOG                  
PDB1_0001   FLEX        STRIPING    COARSE  ONLINELOG                  
PDB1_0001   FLEX        REDUNDANCY  MIRROR  ARCHIVELOG                 
PDB1_0001   FLEX        STRIPING    COARSE  ARCHIVELOG                 
PDB1_0001   FLEX        REDUNDANCY  MIRROR  TEMPFILE                   
PDB1_0001   FLEX        STRIPING    COARSE  TEMPFILE                   
PDB1_0001   FLEX        REDUNDANCY  MIRROR  BACKUPSET                  
PDB1_0001   FLEX        STRIPING    COARSE  BACKUPSET                  
PDB1_0001   FLEX        REDUNDANCY  MIRROR  PARAMETERFILE              
PDB1_0001   FLEX        STRIPING    COARSE  PARAMETERFILE              
PDB1_0001   FLEX        REDUNDANCY  MIRROR  DATAGUARDCONFIG            
PDB1_0001   FLEX        STRIPING    COARSE  DATAGUARDCONFIG            
PDB1_0001   FLEX        REDUNDANCY  MIRROR  CHANGETRACKING             
PDB1_0001   FLEX        STRIPING    COARSE  CHANGETRACKING             
PDB1_0001   FLEX        REDUNDANCY  MIRROR  FLASHBACK                  
PDB1_0001   FLEX        STRIPING    COARSE  FLASHBACK                  
PDB1_0001   FLEX        REDUNDANCY  MIRROR  DUMPSET                    
PDB1_0001   FLEX        STRIPING    COARSE  DUMPSET                    
PDB1_0001   FLEX        REDUNDANCY  MIRROR  AUTOBACKUP                 
PDB1_0001   FLEX        STRIPING    COARSE  AUTOBACKUP                 
PDB1_0001   FLEX        REDUNDANCY  MIRROR  VOTINGFILE                 
PDB1_0001   FLEX        STRIPING    COARSE  VOTINGFILE                 
PDB1_0001   FLEX        REDUNDANCY  MIRROR  OCRFILE                    
PDB1_0001   FLEX        STRIPING    COARSE  OCRFILE                    
PDB1_0001   FLEX        REDUNDANCY  MIRROR  ASMVOL                     
PDB1_0001   FLEX        STRIPING    COARSE  ASMVOL                     
PDB1_0001   FLEX        REDUNDANCY  MIRROR  ASMVDRL                    
PDB1_0001   FLEX        STRIPING    COARSE  ASMVDRL                    
PDB1_0001   FLEX        REDUNDANCY  MIRROR  OCRBACKUP                  
PDB1_0001   FLEX        STRIPING    COARSE  OCRBACKUP                  
PDB1_0001   FLEX        REDUNDANCY  MIRROR  FLASHFILE                  
PDB1_0001   FLEX        STRIPING    COARSE  FLASHFILE                  
PDB1_0001   FLEX        REDUNDANCY  MIRROR  XTRANSPORT BACKUPSET       
PDB1_0001   FLEX        STRIPING    COARSE  XTRANSPORT BACKUPSET       
PDB1_0001   FLEX        REDUNDANCY  MIRROR  AUDIT_SPILLFILES           
PDB1_0001   FLEX        STRIPING    COARSE  AUDIT_SPILLFILES           
PDB1_0001   FLEX        REDUNDANCY  MIRROR  INCR XTRANSPORT BACKUPSET  
PDB1_0001   FLEX        STRIPING    COARSE  INCR XTRANSPORT BACKUPSET  
PDB1_0001   FLEX        REDUNDANCY  MIRROR  KEY_STORE                  
PDB1_0001   FLEX        STRIPING    COARSE  KEY_STORE                  
PDB1_0001   FLEX        REDUNDANCY  MIRROR  AUTOLOGIN_KEY_STORE        
PDB1_0001   FLEX        STRIPING    COARSE  AUTOLOGIN_KEY_STORE        
PDB1_0001   FLEX        REDUNDANCY  MIRROR  CONTAINER                  
PDB1_0001   FLEX        REDUNDANCY  HIGH    CONTROLFILE                
ASMCMD> 

This output shows 2 properties per ASM-supported file type: redundancy and striping. I really only care about redundancy, and I haven’t ever touched the striping property. Quoting the ASM Administrator’s guide, chapter Administering Oracle ASM Disk Groups, section Managing Oracle ASM Flex Disk Groups about the STRIPING property:

This is a file type property, and is set for each file type. Usually the default value for each file type is sufficient and is not changed

I’m happy to go with that.

The SQL interface can provide the same information, and here is the equivalent output:

SQL> select file_type, name, value from v$asm_filegroup_property where filegroup_number = 6;

FILE_TYPE                      NAME                           VALUE
------------------------------ ------------------------------ ------------------------------
                               PRIORITY                       MEDIUM
CONTROLFILE                    REDUNDANCY                     HIGH
CONTROLFILE                    STRIPING                       FINE
DATAFILE                       REDUNDANCY                     MIRROR
DATAFILE                       STRIPING                       COARSE
ONLINELOG                      REDUNDANCY                     MIRROR
ONLINELOG                      STRIPING                       COARSE
ARCHIVELOG                     REDUNDANCY                     MIRROR
ARCHIVELOG                     STRIPING                       COARSE
TEMPFILE                       REDUNDANCY                     MIRROR
TEMPFILE                       STRIPING                       COARSE
BACKUPSET                      REDUNDANCY                     MIRROR
BACKUPSET                      STRIPING                       COARSE
PARAMETERFILE                  REDUNDANCY                     MIRROR
PARAMETERFILE                  STRIPING                       COARSE
DATAGUARDCONFIG                REDUNDANCY                     MIRROR
DATAGUARDCONFIG                STRIPING                       COARSE
CHANGETRACKING                 REDUNDANCY                     MIRROR
CHANGETRACKING                 STRIPING                       COARSE
FLASHBACK                      REDUNDANCY                     MIRROR
FLASHBACK                      STRIPING                       COARSE
DUMPSET                        REDUNDANCY                     MIRROR
DUMPSET                        STRIPING                       COARSE
AUTOBACKUP                     REDUNDANCY                     MIRROR
AUTOBACKUP                     STRIPING                       COARSE
VOTINGFILE                     REDUNDANCY                     MIRROR
VOTINGFILE                     STRIPING                       COARSE
OCRFILE                        REDUNDANCY                     MIRROR
OCRFILE                        STRIPING                       COARSE
ASMVOL                         REDUNDANCY                     MIRROR
ASMVOL                         STRIPING                       COARSE
ASMVDRL                        REDUNDANCY                     MIRROR
ASMVDRL                        STRIPING                       COARSE
OCRBACKUP                      REDUNDANCY                     MIRROR
OCRBACKUP                      STRIPING                       COARSE
FLASHFILE                      REDUNDANCY                     MIRROR
FLASHFILE                      STRIPING                       COARSE
XTRANSPORT BACKUPSET           REDUNDANCY                     MIRROR
XTRANSPORT BACKUPSET           STRIPING                       COARSE
AUDIT_SPILLFILES               REDUNDANCY                     MIRROR
AUDIT_SPILLFILES               STRIPING                       COARSE
INCR XTRANSPORT BACKUPSET      REDUNDANCY                     MIRROR
INCR XTRANSPORT BACKUPSET      STRIPING                       COARSE
KEY_STORE                      REDUNDANCY                     MIRROR
KEY_STORE                      STRIPING                       COARSE
AUTOLOGIN_KEY_STORE            REDUNDANCY                     MIRROR
AUTOLOGIN_KEY_STORE            STRIPING                       COARSE
CONTAINER                      REDUNDANCY                     MIRROR
CONTAINER                      STRIPING                       COARSE

49 rows selected.

In addition to v$asm_file you can also query the new view v$asm_filegroup_file for information about files in File Groups:

SQL> select filegroup_number, file_number, incarnation
  2  from v$asm_filegroup_file
  3  where filegroup_number = 6
  4  order by file_number;

FILEGROUP_NUMBER FILE_NUMBER INCARNATION
---------------- ----------- -----------
               6         309   948464269
               6         310   948464269
               6         311   948464269
               6         312   948464283

SQL> 

FILE_NUMBER and INCARNATION can be used to link back to v$asm_file by the way.

Back to the blog post: I wanted to increase the redundancy level from normal redundancy to high redundancy, but only within filegroup 6. There are a number of useful attributes in v$asm_file that provide information about the size, type, current redundancy, stripe levels and whether a re-mirror operation is taking place.

Before making any changes, this is what is looks like:

SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
  2  from v$asm_file where filegroup_number = 6;

FILE_NUMBER      BYTES      SPACE TYPE                 REDUND R STRIPE R
----------- ---------- ---------- -------------------- ------ - ------ -
        309  104865792  218103808 DATAFILE             MIRROR U COARSE N
        310  262152192  541065216 DATAFILE             MIRROR U COARSE N
        311  419438592  859832320 DATAFILE             MIRROR U COARSE N
        312   67117056  142606336 TEMPFILE             MIRROR U COARSE N

SQL>

Files 309 through 312 use a redundancy of MIRROR, which is 2-way mirroring aka normal redundancy. Let’s try and change this and see what happens.

Altering the datafile.redundancy

This is where the action starts for real. All the filegroup properties that can be changed are documented in the Automatic Storage Management Administrator’s Guide, Administering Oracle ASM Disk Groups chapter, section Managing Oracle ASM Flex Disk Groups. Using the documented example, I can change the redundancy of my data files in filegroup 6:

SQL> alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy'='high';

Diskgroup altered.

This command, like all others that change the properties of storage, must be executed from the ASM instance as SYSASM, or else it fails as shown here when I tried to execute if from the RDBMS instance.

SQL> alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy' = 'high';

Error starting at line : 1 in command -
alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy' = 'high'
Error report -
ORA-15000: command disallowed by current instance type
15000. 00000 -  "command disallowed by current instance type"
*Cause:    The user has issued a command to a conventional RDBMS instance
           that is only appropriate for an ASM instance. Alternatively, the
           user has issued a command to an ASM instance that is only
           appropriate for an RDBMS instance.
*Action:   Connect to the correct instance type and re-issue the command.
SQL> 

Immediately after this command completes data files in filegroup 6 are re-mirrored. While the re-mirror operation is ongoing (see flag remirror=’Y’) the redundancy is still set to a value of MIRROR. Only when it has completed (remirror = ‘N’) is the redundancy changed to HIGH. Note that file 312 in the filegroup still is set to NORMAL redundancy. On second look you’ll notice it’s ok because it is a tempfile, not a data file. And I only asked for datafiles to be protected by high redundancy.

SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
  2  from v$asm_file where filegroup_number = 6;

FILE_NUMBER      BYTES      SPACE TYPE                 REDUND R STRIPE R
----------- ---------- ---------- -------------------- ------ - ------ -
        309  104865792  339738624 DATAFILE             MIRROR U COARSE Y
        310  262152192  805306368 DATAFILE             MIRROR U COARSE Y
        311  419438592 1283457024 DATAFILE             MIRROR U COARSE Y
        312   67117056  142606336 TEMPFILE             MIRROR U COARSE N

SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
  2  from v$asm_file where filegroup_number = 6;

FILE_NUMBER      BYTES      SPACE TYPE                 REDUND R STRIPE R
----------- ---------- ---------- -------------------- ------ - ------ -
        309  104865792  339738624 DATAFILE             HIGH   U COARSE N
        310  262152192  805306368 DATAFILE             HIGH   U COARSE N
        311  419438592 1283457024 DATAFILE             HIGH   U COARSE N
        312   67117056  142606336 TEMPFILE             MIRROR U COARSE N

To complete the post I’ll list the ASM instance’s alert.log as it’s quite verbose and I find it interesting to see what activities ASM performs.

SQL> alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy'='high'
NOTE: updated format of group 5 file 311 for 3-way mirroring
NOTE: updated redundancy of group 5 file 311 to 3-way mirrored (remirror 0x4)
NOTE: updated format of group 5 file 310 for 3-way mirroring
NOTE: updated redundancy of group 5 file 310 to 3-way mirrored (remirror 0x4)
NOTE: updated format of group 5 file 309 for 3-way mirroring
NOTE: updated redundancy of group 5 file 309 to 3-way mirrored (remirror 0x4)
NOTE: GroupBlock outside rolling migration privileged region
NOTE: client +ASM1:+ASM:rac122pri no longer has group 5 (FLEX) mounted
NOTE: client +ASM1:+ASM:rac122pri no longer has group 3 (DATA) mounted
NOTE: client +ASM1:+ASM:rac122pri no longer has group 2 (MGMT) mounted
NOTE: requesting all-instance membership refresh for group=5
NOTE: membership refresh pending for group 5/0x4718f00c (FLEX)
GMON querying group 5 at 835 for pid 25, osid 11576
SUCCESS: refreshed membership for 5/0x4718f00c (FLEX)
SUCCESS: alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy'='high'
2017-07-06 13:17:47.169000 +01:00
NOTE: Attempting voting file refresh on diskgroup FLEX
NOTE: Refresh completed on diskgroup FLEX. No voting file found.
NOTE: starting rebalance of group 5/0x4718f00c (FLEX) at power 1
NOTE: starting process ARBA
Starting background process ARBA
ARBA started with pid=33, OS id=9904
NOTE: starting process ARB0
Starting background process ARB0
ARB0 started with pid=48, OS id=9906
NOTE: assigning ARBA to group 5/0x4718f00c (FLEX) to compute estimates
NOTE: assigning ARB0 to group 5/0x4718f00c (FLEX) with 1 parallel I/O
2017-07-06 13:18:29.554000 +01:00
NOTE: Starting expel slave for group 5/0x4718f00c (FLEX)
NOTE: stopping process ARB0
NOTE: stopping process ARBA
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=5
SUCCESS: rebalance completed for group 5/0x4718f00c (FLEX)
NOTE: membership refresh pending for group 5/0x4718f00c (FLEX)
GMON querying group 5 at 836 for pid 25, osid 11576
SUCCESS: refreshed membership for 5/0x4718f00c (FLEX)
2017-07-06 13:18:32.568000 +01:00
NOTE: Attempting voting file refresh on diskgroup FLEX
NOTE: Refresh completed on diskgroup FLEX. No voting file found.

The re-mirroring operation is reported as a (mini) rebalance operation in the ASM instance’s alert.log. Thinking about it this makes perfect sense.

What about new files?

The change in meta-data is also visible in v$asm_filegroup_properties. Every new data file created in my PDB will from now on be created with high redundancy.

SQL> select file_type, name, value from v$asm_filegroup_property
  2  where filegroup_number = 6 and file_type = 'DATAFILE';

FILE_TYPE       NAME                 VALUE
--------------- -------------------- --------------------
DATAFILE        REDUNDANCY           HIGH
DATAFILE        STRIPING             COARSE

And indeed: after adding a USERS-tablespace the new properties are enforced:

SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
  2  from v$asm_file where filegroup_number = 6
  3  order by file_number;

FILE_NUMBER      BYTES      SPACE TYPE            REDUND R STRIPE R
----------- ---------- ---------- --------------- ------ - ------ -
        309  167780352  528482304 DATAFILE        HIGH   U COARSE N
        310  272637952  843055104 DATAFILE        HIGH   U COARSE N
        311  471867392 1434451968 DATAFILE        HIGH   U COARSE N
        312   67117056  142606336 TEMPFILE        MIRROR U COARSE N
        313 2147491840 6467616768 DATAFILE        HIGH   U COARSE N

Summary

Flex ASM Disk Groups continue to amaze me. Using the File Group properties I can now define entities (NCDB, CDB, PDB, …) and provide flexible, highly granular settings for data protection within the disk group. In previous releases I could do similar things, but that required multiple disk groups and was more complex to do so.

@martinberx posted an interesting comment about Flex ASM Disk Groups and redundancy levels on twitter, comparing Flex ASM Disk Groups with ASM File Templates. Using ASM File Templates one could define redundancy levels of files within an ASM disk group amongst other things, a feature which has been with ASM for a long time. Technically it is possible to define a template for data files mandating 3-way mirroring even if the file was stored on a normal redundancy disk group.

My colleague Alex Fatkulin has blogged about high redundancy files in normal redundancy disk groups and shows why ASM File Templates ultimately don’t offer better availability. I’ll try and get a test case together for my Flex ASM Disk Group to see if it fares better. According to the documentation, a Flex ASM Disk Group with 5 or more failgroups should be able to survive the failure of 2 failgroups (just like a disk group with High Redundancy).

Advertisements

One thought on “12.2 New Feature: the FLEX ASM disk group part 4

  1. Pingback: Log Buffer #519: A Carnival of the Vanities for DBAs – Cloud Data Architect

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s