How To Fix ASM Disk Header Status from “FORMER” To “MEMBER”

Posted in: Technical Track

I made an error when creating an Oracle ASM disk group which resulted in not being able to drop or mount the disk group because the ASM header_status was FORMER.

The following steps were performed to modify the ASM header_status from FORMER to MEMBER for version 12.1.0.2.0.

Mounting and dropping disk group failed.

SQL> drop diskgroup ACFS_DATA;
drop diskgroup ACFS_DATA
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15001: diskgroup "ACFS_DATA" does not exist or is not mounted


SQL> alter diskgroup ACFS_DATA mount;
alter diskgroup ACFS_DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "ACFS_DATA" cannot be mounted
ORA-15040: diskgroup is incomplete

SQL> 

Review ASM disks.
Note header_status is FORMER and group# is 0 (does not belong to group).

SQL> SELECT group_number group#, disk_number disk#, header_status header_, mode_status mode_, state, library, os_mb, total_mb, free_mb, label, path
  2  FROM V$ASM_DISK
  3  ORDER BY header_status desc, path
  4  ;

GROUP# DISK# HEADER_    MODE_      STATE        LIBRARY                           OS_MB   TOTAL_MB    FREE_MB LABEL           PATH
------ ----- ---------- ---------- ------------ ---------------------------- ---------- ---------- ---------- --------------- ----------------------------------------
     0     1 FORMER     ONLINE     NORMAL       System                           153584          0          0                 /dev/mapper/acfs_01p1

22 rows selected.

Let’s fix the issue.

Documentation used head -25 and might be better to grep for kfdhdb.hdrsts

### Use kfed to read device and confirm header_status KFDHDR_FORMER

$ kfed read '/dev/mapper/acfs_01p1' | head -25
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 1999457674 ; 0x00c: 0x772d4d8a
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 202375168 ; 0x020: 0x0c100000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
**********************************************************************
kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER
**********************************************************************
kfdhdb.dskname: ACFS_DATA_0000 ; 0x028: length=14
kfdhdb.grpname: ACFS_DATA ; 0x048: length=9
kfdhdb.fgname: ACFS_DATA_0000 ; 0x068: length=14

### Backup device
$ mkdir -p /tmp/asm
$ dd if=/dev/mapper/acfs_01p1 of=/tmp/asm/ihotelp1_acfs.dmp bs=1048576 count=50
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.303025 s, 173 MB/s

### Create template for patching device
$ kfed read /dev/mapper/acfs_01p1 > /tmp/asm/patch_acfs_01p1.txt

### Create copy of template for comparison
$ cp patch_acfs_01p1.txt patch_acfs_01p1.orig

### Edit patch_acfs_01p1.txt and change "kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER" to "kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER"

### Verify change.
$ diff patch_acfs_01p1.txt patch_acfs_01p1.orig
< kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER --- > kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER

### Patch device acfs_01p1 using patch_acfs_01p1.txt
$ kfed merge /dev/mapper/acfs_01p1 text=patch_acfs_01p1.txt

### Verify device acfs_01p1 is updated`
$ kfed read /dev/mapper/acfs_01p1 | grep kfdhdb.hdrsts
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
$

Verify results using SQL

SQL> SELECT group_number group#, disk_number disk#, header_status header_, mode_status mode_, state, library, os_mb, total_mb, free_mb, label, path
  2  FROM V$ASM_DISK
  3  ORDER BY header_status desc, path
  4  ;

GROUP# DISK# HEADER_    MODE_      STATE        LIBRARY                           OS_MB   TOTAL_MB    FREE_MB LABEL           PATH
------ ----- ---------- ---------- ------------ ---------------------------- ---------- ---------- ---------- --------------- ----------------------------------------
     0     1 MEMBER     ONLINE     NORMAL       System                           153584          0          0                 /dev/mapper/acfs_01p1

SQL> select name from v$asm_diskgroup order by 1;

NAME
------------------------------------------------------------------------------------------
ACFS_DATA

SQL> alter diskgroup ACFS_DATA mount;

Diskgroup altered.

SQL> SELECT group_number group#, disk_number disk#, header_status header_, mode_status mode_, state, library, os_mb, total_mb, free_mb, label, path
  2  FROM V$ASM_DISK
  3  ORDER BY header_status desc, path
  4  ;

GROUP# DISK# HEADER_    MODE_      STATE        LIBRARY                           OS_MB   TOTAL_MB    FREE_MB LABEL           PATH
------ ----- ---------- ---------- ------------ ---------------------------- ---------- ---------- ---------- --------------- ----------------------------------------
     5     0 MEMBER     ONLINE     NORMAL       System                           153584     153584     153500                 /dev/mapper/acfs_01p1

SQL> select name from v$asm_diskgroup order by 1;

NAME
------------------------------------------------------------------------------------------
ACFS_DATA

SQL> 

no rows selected

SQL> 

Verify results using asmcmd.

$ asmcmd lsdg -g ACFS_DATA
Inst_ID  State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  EXTERN  N         512   4096  4194304    153584   153500                0          153500              0             N  ACFS_DATA/

$ asmcmd lsattr -l -G ACFS_DATA
Name                     Value
access_control.enabled   FALSE
access_control.umask     066
au_size                  4194304
cell.smart_scan_capable  FALSE
compatible.advm          12.1.0.0.0
compatible.asm           12.1.0.0.0
compatible.rdbms         12.1.0.0.0
content.check            FALSE
content.type             data
disk_repair_time         3.6h
failgroup_repair_time    24.0h
idp.boundary             auto
idp.type                 dynamic
phys_meta_replicated     true
sector_size              512
thin_provisioned         FALSE
$

Hopefully the information and demonstration will be useful should you encounter the same issues.

Reference:
ASM Corruption: Case #1: How To Fix The ASM Disk HEADER_STATUS From FORMER or PROVISIONED To MEMBER. (Doc ID 1448799.1)

email

Interested in working with Michael? Schedule a tech call.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *