Flex Disk Group Properties
In the previous 3 parts I shared my investigation into ASM Flex Disk Groups, Quota Groups, File Groups, and how Quota Groups actually enforce space limits. What I haven’t discussed yet was changing properties of a File Group and the effects thereof. Properties I have in mind are related to the protection level, as discussed in the official documentation-Automatic Storage Management Administrator’s Guide, Administering Oracle ASM Disk Groups. There are of course other properties as well (and you’ll find a link to all of the modifiable properties later in this post), but they are out of scope for this investigation.
A disk group with Flex Redundancy can be set up with all protection levels (3-way mirror, 2-way mirror, unprotected), and by default uses 2-way mirroring. Unlike other types of disk groups, you can change the protection levels for individual (pluggable) databases within the Flex Disk group. This is best shown with an example. Continuing the story from my previous blog posts, here’s the setup again for your convenience.
SQL> select filegroup_number,name,guid from v$asm_filegroup
FILEGROUP_NUMBER NAME GUID
---------------- -------------------- --------------------------------
0 DEFAULT_FILEGROUP
1 CDB_CDB$ROOT 4700A987085A3DFAE05387E5E50A8C7B
2 CDB_PDB$SEED 536DF51E8E28221BE0534764A8C0FD81
3 PDB1 537B677EF8DA0F1AE0534764A8C05729
4 ORCL_CDB$ROOT 4700A987085A3DFAE05387E5E50A8C7B
5 ORCL_PDB$SEED 537E63B952183748E0534764A8C09A7F
6 PDB1_0001 537EB5B87E62586EE0534764A8C05530
7 rows selected.
The above listing shows all my File Groups in my ASM instance. This post is about changing attributes of filegroup number 6, PDB1_0001. It will be important to understand which files pertain to the filegroup later; here’s the list:
SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror 2 from v$asm_file where filegroup_number = 6; FILE_NUMBER BYTES SPACE TYPE REDUNDANCY REDUNDANCY_LOWERED STRIPED REMIRROR 309 104865792 218103808 DATAFILE MIRROR U COARSE N 310 262152192 541065216 DATAFILE MIRROR U COARSE N 311 419438592 859832320 DATAFILE MIRROR U COARSE N 312 67117056 142606336 TEMPFILE MIRROR U COARSE N
The documentation is correct: the default redundancy for the datafiles and tempfile is “mirror”.
Filegroup properties
The ability to change redundancy and other properties within the disk group hinges on the fact that you have File Groups. Properties that belong to a File Group can be listed either via the SQL interface, or asmcmd. The latter is shown first:
ASMCMD> lsfg -G flex --filegroup PDB1_0001 File Group Disk Group Property Value File Type PDB1_0001 FLEX PRIORITY MEDIUM PDB1_0001 FLEX STRIPING COARSE CONTAINER PDB1_0001 FLEX STRIPING FINE CONTROLFILE PDB1_0001 FLEX REDUNDANCY MIRROR DATAFILE PDB1_0001 FLEX STRIPING COARSE DATAFILE PDB1_0001 FLEX REDUNDANCY MIRROR ONLINELOG PDB1_0001 FLEX STRIPING COARSE ONLINELOG PDB1_0001 FLEX REDUNDANCY MIRROR ARCHIVELOG PDB1_0001 FLEX STRIPING COARSE ARCHIVELOG PDB1_0001 FLEX REDUNDANCY MIRROR TEMPFILE PDB1_0001 FLEX STRIPING COARSE TEMPFILE PDB1_0001 FLEX REDUNDANCY MIRROR BACKUPSET PDB1_0001 FLEX STRIPING COARSE BACKUPSET PDB1_0001 FLEX REDUNDANCY MIRROR PARAMETERFILE PDB1_0001 FLEX STRIPING COARSE PARAMETERFILE PDB1_0001 FLEX REDUNDANCY MIRROR DATAGUARDCONFIG PDB1_0001 FLEX STRIPING COARSE DATAGUARDCONFIG PDB1_0001 FLEX REDUNDANCY MIRROR CHANGETRACKING PDB1_0001 FLEX STRIPING COARSE CHANGETRACKING PDB1_0001 FLEX REDUNDANCY MIRROR FLASHBACK PDB1_0001 FLEX STRIPING COARSE FLASHBACK PDB1_0001 FLEX REDUNDANCY MIRROR DUMPSET PDB1_0001 FLEX STRIPING COARSE DUMPSET PDB1_0001 FLEX REDUNDANCY MIRROR AUTOBACKUP PDB1_0001 FLEX STRIPING COARSE AUTOBACKUP PDB1_0001 FLEX REDUNDANCY MIRROR VOTINGFILE PDB1_0001 FLEX STRIPING COARSE VOTINGFILE PDB1_0001 FLEX REDUNDANCY MIRROR OCRFILE PDB1_0001 FLEX STRIPING COARSE OCRFILE PDB1_0001 FLEX REDUNDANCY MIRROR ASMVOL PDB1_0001 FLEX STRIPING COARSE ASMVOL PDB1_0001 FLEX REDUNDANCY MIRROR ASMVDRL PDB1_0001 FLEX STRIPING COARSE ASMVDRL PDB1_0001 FLEX REDUNDANCY MIRROR OCRBACKUP PDB1_0001 FLEX STRIPING COARSE OCRBACKUP PDB1_0001 FLEX REDUNDANCY MIRROR FLASHFILE PDB1_0001 FLEX STRIPING COARSE FLASHFILE PDB1_0001 FLEX REDUNDANCY MIRROR XTRANSPORT BACKUPSET PDB1_0001 FLEX STRIPING COARSE XTRANSPORT BACKUPSET PDB1_0001 FLEX REDUNDANCY MIRROR AUDIT_SPILLFILES PDB1_0001 FLEX STRIPING COARSE AUDIT_SPILLFILES PDB1_0001 FLEX REDUNDANCY MIRROR INCR XTRANSPORT BACKUPSET PDB1_0001 FLEX STRIPING COARSE INCR XTRANSPORT BACKUPSET PDB1_0001 FLEX REDUNDANCY MIRROR KEY_STORE PDB1_0001 FLEX STRIPING COARSE KEY_STORE PDB1_0001 FLEX REDUNDANCY MIRROR AUTOLOGIN_KEY_STORE PDB1_0001 FLEX STRIPING COARSE AUTOLOGIN_KEY_STORE PDB1_0001 FLEX REDUNDANCY MIRROR CONTAINER PDB1_0001 FLEX REDUNDANCY HIGH CONTROLFILE ASMCMD>
This output shows 2 properties per ASM-supported file type: redundancy and striping. I really only care about redundancy, and I haven’t ever touched the striping property. Quoting the ASM Administrator’s guide, chapter Administering Oracle ASM Disk Groups, section Managing Oracle ASM Flex Disk Groups about the STRIPING property:
This is a file type property, and is set for each file type. Usually the default value for each file type is sufficient and is not changed
I’m happy to go with that.
The SQL interface can provide the same information, and here is the equivalent output:
SQL> select file_type, name, value from v$asm_filegroup_property where filegroup_number = 6;
FILE_TYPE NAME VALUE
------------------------------ ------------------------------ ------------------------------
PRIORITY MEDIUM
CONTROLFILE REDUNDANCY HIGH
CONTROLFILE STRIPING FINE
DATAFILE REDUNDANCY MIRROR
DATAFILE STRIPING COARSE
ONLINELOG REDUNDANCY MIRROR
ONLINELOG STRIPING COARSE
ARCHIVELOG REDUNDANCY MIRROR
ARCHIVELOG STRIPING COARSE
TEMPFILE REDUNDANCY MIRROR
TEMPFILE STRIPING COARSE
BACKUPSET REDUNDANCY MIRROR
BACKUPSET STRIPING COARSE
PARAMETERFILE REDUNDANCY MIRROR
PARAMETERFILE STRIPING COARSE
DATAGUARDCONFIG REDUNDANCY MIRROR
DATAGUARDCONFIG STRIPING COARSE
CHANGETRACKING REDUNDANCY MIRROR
CHANGETRACKING STRIPING COARSE
FLASHBACK REDUNDANCY MIRROR
FLASHBACK STRIPING COARSE
DUMPSET REDUNDANCY MIRROR
DUMPSET STRIPING COARSE
AUTOBACKUP REDUNDANCY MIRROR
AUTOBACKUP STRIPING COARSE
VOTINGFILE REDUNDANCY MIRROR
VOTINGFILE STRIPING COARSE
OCRFILE REDUNDANCY MIRROR
OCRFILE STRIPING COARSE
ASMVOL REDUNDANCY MIRROR
ASMVOL STRIPING COARSE
ASMVDRL REDUNDANCY MIRROR
ASMVDRL STRIPING COARSE
OCRBACKUP REDUNDANCY MIRROR
OCRBACKUP STRIPING COARSE
FLASHFILE REDUNDANCY MIRROR
FLASHFILE STRIPING COARSE
XTRANSPORT BACKUPSET REDUNDANCY MIRROR
XTRANSPORT BACKUPSET STRIPING COARSE
AUDIT_SPILLFILES REDUNDANCY MIRROR
AUDIT_SPILLFILES STRIPING COARSE
INCR XTRANSPORT BACKUPSET REDUNDANCY MIRROR
INCR XTRANSPORT BACKUPSET STRIPING COARSE
KEY_STORE REDUNDANCY MIRROR
KEY_STORE STRIPING COARSE
AUTOLOGIN_KEY_STORE REDUNDANCY MIRROR
AUTOLOGIN_KEY_STORE STRIPING COARSE
CONTAINER REDUNDANCY MIRROR
CONTAINER STRIPING COARSE
49 rows selected.
In addition to v$asm_file you can also query the new view v$asm_filegroup_file for information about files in File Groups:
SQL> select filegroup_number, file_number, incarnation
2 from v$asm_filegroup_file
3 where filegroup_number = 6
4 order by file_number;
FILEGROUP_NUMBER FILE_NUMBER INCARNATION
---------------- ----------- -----------
6 309 948464269
6 310 948464269
6 311 948464269
6 312 948464283
SQL>
FILE_NUMBER and INCARNATION can be used to link back to v$asm_file by the way.
Back to the blog post: I wanted to increase the redundancy level from normal redundancy to high redundancy, but only within filegroup 6. There are a number of useful attributes in v$asm_file that provide information about the size, type, current redundancy, stripe levels and whether a re-mirror operation is taking place.
Before making any changes, this is what is looks like:
SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
2 from v$asm_file where filegroup_number = 6;
FILE_NUMBER BYTES SPACE TYPE REDUND R STRIPE R
----------- ---------- ---------- -------------------- ------ - ------ -
309 104865792 218103808 DATAFILE MIRROR U COARSE N
310 262152192 541065216 DATAFILE MIRROR U COARSE N
311 419438592 859832320 DATAFILE MIRROR U COARSE N
312 67117056 142606336 TEMPFILE MIRROR U COARSE N
SQL>
Files 309 through 312 use a redundancy of MIRROR, which is 2-way mirroring aka normal redundancy. Let’s try and change this and see what happens.
Altering the datafile.redundancy
This is where the action starts for real. All the filegroup properties that can be changed are documented in the Automatic Storage Management Administrator’s Guide, Administering Oracle ASM Disk Groups chapter, section Managing Oracle ASM Flex Disk Groups. Using the documented example, I can change the redundancy of my data files in filegroup 6:
SQL> alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy'='high'; Diskgroup altered.
This command, like all others that change the properties of storage, must be executed from the ASM instance as SYSASM, or else it fails as shown here when I tried to execute if from the RDBMS instance.
SQL> alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy' = 'high';
Error starting at line : 1 in command -
alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy' = 'high'
Error report -
ORA-15000: command disallowed by current instance type
15000. 00000 - "command disallowed by current instance type"
*Cause: The user has issued a command to a conventional RDBMS instance
that is only appropriate for an ASM instance. Alternatively, the
user has issued a command to an ASM instance that is only
appropriate for an RDBMS instance.
*Action: Connect to the correct instance type and re-issue the command.
SQL>
Immediately after this command completes data files in filegroup 6 are re-mirrored. While the re-mirror operation is ongoing (see flag remirror=’Y’) the redundancy is still set to a value of MIRROR. Only when it has completed (remirror = ‘N’) is the redundancy changed to HIGH. Note that file 312 in the filegroup still is set to NORMAL redundancy. On second look you’ll notice it’s ok because it is a tempfile, not a data file. And I only asked for datafiles to be protected by high redundancy.
SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
2 from v$asm_file where filegroup_number = 6;
FILE_NUMBER BYTES SPACE TYPE REDUND R STRIPE R
----------- ---------- ---------- -------------------- ------ - ------ -
309 104865792 339738624 DATAFILE MIRROR U COARSE Y
310 262152192 805306368 DATAFILE MIRROR U COARSE Y
311 419438592 1283457024 DATAFILE MIRROR U COARSE Y
312 67117056 142606336 TEMPFILE MIRROR U COARSE N
SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
2 from v$asm_file where filegroup_number = 6;
FILE_NUMBER BYTES SPACE TYPE REDUND R STRIPE R
----------- ---------- ---------- -------------------- ------ - ------ -
309 104865792 339738624 DATAFILE HIGH U COARSE N
310 262152192 805306368 DATAFILE HIGH U COARSE N
311 419438592 1283457024 DATAFILE HIGH U COARSE N
312 67117056 142606336 TEMPFILE MIRROR U COARSE N
To complete the post I’ll list the ASM instance’s alert.log as it’s quite verbose and I find it interesting to see what activities ASM performs.
SQL> alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy'='high' NOTE: updated format of group 5 file 311 for 3-way mirroring NOTE: updated redundancy of group 5 file 311 to 3-way mirrored (remirror 0x4) NOTE: updated format of group 5 file 310 for 3-way mirroring NOTE: updated redundancy of group 5 file 310 to 3-way mirrored (remirror 0x4) NOTE: updated format of group 5 file 309 for 3-way mirroring NOTE: updated redundancy of group 5 file 309 to 3-way mirrored (remirror 0x4) NOTE: GroupBlock outside rolling migration privileged region NOTE: client +ASM1:+ASM:rac122pri no longer has group 5 (FLEX) mounted NOTE: client +ASM1:+ASM:rac122pri no longer has group 3 (DATA) mounted NOTE: client +ASM1:+ASM:rac122pri no longer has group 2 (MGMT) mounted NOTE: requesting all-instance membership refresh for group=5 NOTE: membership refresh pending for group 5/0x4718f00c (FLEX) GMON querying group 5 at 835 for pid 25, osid 11576 SUCCESS: refreshed membership for 5/0x4718f00c (FLEX) SUCCESS: alter diskgroup flex modify filegroup PDB1_0001 set 'datafile.redundancy'='high' 2017-07-06 13:17:47.169000 +01:00 NOTE: Attempting voting file refresh on diskgroup FLEX NOTE: Refresh completed on diskgroup FLEX. No voting file found. NOTE: starting rebalance of group 5/0x4718f00c (FLEX) at power 1 NOTE: starting process ARBA Starting background process ARBA ARBA started with pid=33, OS id=9904 NOTE: starting process ARB0 Starting background process ARB0 ARB0 started with pid=48, OS id=9906 NOTE: assigning ARBA to group 5/0x4718f00c (FLEX) to compute estimates NOTE: assigning ARB0 to group 5/0x4718f00c (FLEX) with 1 parallel I/O 2017-07-06 13:18:29.554000 +01:00 NOTE: Starting expel slave for group 5/0x4718f00c (FLEX) NOTE: stopping process ARB0 NOTE: stopping process ARBA NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=5 SUCCESS: rebalance completed for group 5/0x4718f00c (FLEX) NOTE: membership refresh pending for group 5/0x4718f00c (FLEX) GMON querying group 5 at 836 for pid 25, osid 11576 SUCCESS: refreshed membership for 5/0x4718f00c (FLEX) 2017-07-06 13:18:32.568000 +01:00 NOTE: Attempting voting file refresh on diskgroup FLEX NOTE: Refresh completed on diskgroup FLEX. No voting file found.
The re-mirroring operation is reported as a (mini) rebalance operation in the ASM instance’s alert.log. Thinking about it this makes perfect sense.
What about new files?
The change in meta-data is also visible in v$asm_filegroup_properties. Every new data file created in my PDB will from now on be created with high redundancy.
SQL> select file_type, name, value from v$asm_filegroup_property 2 where filegroup_number = 6 and file_type = 'DATAFILE'; FILE_TYPE NAME VALUE --------------- -------------------- -------------------- DATAFILE REDUNDANCY HIGH DATAFILE STRIPING COARSE
And indeed: after adding a USERS-tablespace the new properties are enforced:
SQL> select file_number,bytes,space,type,redundancy,redundancy_lowered,striped,remirror
2 from v$asm_file where filegroup_number = 6
3 order by file_number;
FILE_NUMBER BYTES SPACE TYPE REDUND R STRIPE R
----------- ---------- ---------- --------------- ------ - ------ -
309 167780352 528482304 DATAFILE HIGH U COARSE N
310 272637952 843055104 DATAFILE HIGH U COARSE N
311 471867392 1434451968 DATAFILE HIGH U COARSE N
312 67117056 142606336 TEMPFILE MIRROR U COARSE N
313 2147491840 6467616768 DATAFILE HIGH U COARSE N
Summary
Flex ASM Disk Groups continue to amaze me. Using the File Group properties I can now define entities (NCDB, CDB, PDB, …) and provide flexible, highly granular settings for data protection within the disk group. In previous releases I could do similar things, but that required multiple disk groups and was more complex to do so.
@martinberx posted an interesting comment about Flex ASM Disk Groups and redundancy levels on twitter, comparing Flex ASM Disk Groups with ASM File Templates. Using ASM File Templates one could define redundancy levels of files within an ASM disk group amongst other things, a feature which has been with ASM for a long time. Technically it is possible to define a template for data files mandating 3-way mirroring even if the file was stored on a normal redundancy disk group.
My colleague Alex Fatkulin has blogged about high redundancy files in normal redundancy disk groups and shows why ASM File Templates ultimately don’t offer better availability. I’ll try and get a test case together for my Flex ASM Disk Group to see if it fares better. According to the documentation, a Flex ASM Disk Group with 5 or more failgroups should be able to survive the failure of 2 failgroups (just like a disk group with High Redundancy).
Martins Blog 
Pingback: Log Buffer #519: A Carnival of the Vanities for DBAs – Cloud Data Architect