The views expressed on this blog are my own and do not necessarily reflect the views of Oracle

April 26, 2010

kfed - ASM metadata editor


The kfed is an undocumented ASM utility that can be used to read and modify ASM metadata. The kfed does not depend on the mount state of ASM instance or ASM disk group, so it can be used with ASM instance down and on a disk group that does not mount. In fact kfed can be used on ASM disks with corrupt ASM metadata.

Oracle Support may use kfed to correct some problems with damaged ASM metadata. You may want to use kfed to read from ASM disks - e.g. for educational purposes or to check the ASM metadata.

If you don't see the kfed binary in your $ORACLE_HOME/bin directory (in ASM /Grid Infrastructure home), you can build as follows:

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins* ikfed

Check if ASM disk header is healthy

The following is an example of using kfed to read the contents of the ASM disk header from ASMLIB disk DISK4 (I left out the bits of no immediate interest).

$ kfed read /dev/oracleasm/disks/DISK4
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
...
kfbh.check: 1539641569 ; 0x00c: 0x5bc510e1
...
kfdhdb.driver.provstr: ORCLDISKDISK4 ; 0x000: length=13
...
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 2 ; 0x026: KFDGTP_NORMAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: PLAY0 ; 0x028: length=5
kfdhdb.grpname: PLAY ; 0x048: length=4
kfdhdb.fgname: P1 ; 0x068: length=2
...
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 4194304 ; 0x0bc: 0x00400000
...
kfdhdb.dsksize: 1221 ; 0x0c4: 0x000004c5
...

Valid ASM disk should have kfbh.type=KFBTYP_DISKHEAD (ASM disk header).

ASMLIB disk name should follow after 'ORCLDISK' in kfdhdb.driver.provstr field. Note that ASMLIB disk name does not have to be the same as ASM disk name.

Is this really the disk with ASM name PLAY0 (kfdhdb.dskname) and disk number 0 (kfdhdb.dsknum=0) in disk group PLAY (kfdhdb.grpname)? If you are not sure, check the ASM alert log entries around the time of the last successful mount of disk group PLAY.

Header status says MEMBER for this disk (kfdhdb.hdrsts=KFDHDR_MEMBER). That is what we want to see.

ASM metadata block size is 4 KB (kfdhdb.blksize=4096) and allocation unit size is 4 MB for this disk (kfdhdb.ausize=4194304) and the disk size is 1221 AUs, i.e 4884 MB. Is that what you think it should be? Is that what you see at the OS level for that device?

If you see kfbh.type=KFBTYP_INVALID in the disk header on a disk you believe belongs to an ASM disk group, that may indicate that the ASM disk header is damaged. But don't jump to conclusions. Are you looking at the right disk? Is this the right disk partition? Can you access that disk via some other name - in a multipath setup? If you are not sure, or if the disk is in fact damaged, log an SR with Oracle Support to check it out.

I should say that the ASM disk header may look fine, but in fact be corrupt. For example the block checksum (kfbh.check) could be wrong in which case that would need to be corrected. Please log an SR with Oracle Support to assist with that problem.

Note that kfed was used with no additional options. Of note is that no allocation unit number and no block number were specified, which means that default values would be used (0 for both). The command used was:

$ kfed read /dev/oracleasm/disks/DISK4

That is equivalent to:

$ kfed read /dev/oracleasm/disks/DISK4 aun=0 blkn=0

Also note that for reading block 0 or allocation unit 0, there is no need to specify the allocation unit size or the block size.

Check if allocation unit 0 has expected ASM metadata

Read from the same disk, but look at different ASM metadata blocks.

Check block 1:

$ kfed read /dev/oracleasm/disks/DISK4 blkn=1 | grep KFBTYP

Expected result is KFBTYP_FREESPC (free space table).

Check the rest of the blocks in allocation unit 0:

ausize=`kfed read DISK4 | grep ausize | tr -s ' ' | cut -d' ' -f2`
blksize=`kfed read DISK4 | grep blksize | tr -s ' ' | cut -d' ' -f2`
let n=$ausize/$blksize

for (( i=2; i<$n; i++ ))
do
  kfed read /dev/oracleasm/disks/DISK4 blkn=$i | grep KFBTYP
done


Expected result for all those blocks is KFBTYP_ALLOCTBL (allocation table).

Again, if you see kfbh.type=KFBTYP_INVALID in any of the results, that may indicate a corrupted ASM metadata block. My advice is the same as above - log an SR with Oracle Support.

I should point out that with the above we only looked at the expected ASM metadata block types. We did not look at the actual metadata block contents. Some ASM metadata block corruptions are indeed with the block contents. Such corruptions are only detected when ASM reads the corrupt block, in which case ASM instance will report ORA-15196 error. You should always log an SR with Oracle Support if you are unfortunate enough to encounter that error.

Check if allocation unit 1 has expected ASM metadata

Read from the same disk, this time look at all but two last blocks in allocation unit 1.

ausize=`kfed read DISK4 | grep ausize | tr -s ' ' | cut -d' ' -f2`
blksize=`kfed read DISK4 | grep blksize | tr -s ' ' | cut -d' ' -f2`
let n=$ausize/$blksize-2

for (( i=0; i<$n; i++ ))
do
  kfed read /dev/oracleasm/disks/DISK4 ausz=$ausize aun=1 blkn=$i | grep KFBTYP
done


Expected result for all these blocks is one of the following: KFBTYP_PST_META, KFBTYP_PST_DTA or KFBTYP_PST_NONE (partnership and status table blocks).

Read the second last block from allocation unit 1.

$ kfed read /dev/oracleasm/disks/DISK4 ausz=$ausize aun=1 blkn=$n | grep KFBTYP

Expected result in ASM version before 11.1.0.7 is one of the PST block types.

Expected result in ASM version 11.1.0.7 and later is KFBTYP_DISKHEAD. Yes, the backup copy of the ASM disk header. Indeed, if the only problem with the disk is the damaged disk header block, this can be easily repaired. But not on your own - please log an SR with Oracle Support to assist with this kind of problem.

Finally, read the last block in allocation unit 1.

$ let n=$n+1
$ kfed read /dev/oracleasm/disks/DISK4 ausz=$ausize aun=1 blkn=$n | grep KFBTYP

Expected result is KFBTYP_HBEAT (heartbeat block).

Note that on the kfed command line, in addition to allocation unit number (aun) I had to specify the allocation unit size (ausz). The ausz parameter is not required for default allocation unit size (1 MB) and for reading from allocation unit 0.

4 comments:

  1. Where is backup header located which can be restored in case disk header is curropted

    ReplyDelete
    Replies
    1. A backup copy of ASM disk header is in the second last block of allocation unit 1. If the allocation unit (AU) size is 1 MB there will be 256 blocks per AU. In that case the copy of the disk header will be in block 254 (note that blocks go from 0 to 255) of AU 1. The command to read that block would be:

      $ kfed read aun=1 blkn=254

      If the AU size is 4 MB, there will be 1024 blocks per AU. In that case the copy of the disk header will be in block 1022 (blocks now go from 0 to 1023) of AU 1. The command to read that block would be:

      $ kfed read ausz=4194304 aun=1 blkn=1022

      Note that this time I had to specify the AU size in the kfed command as AU had a non default size.

      While we are on this topic, if the only problem is the disk header corruption, that can easily be repaired with 'kfed repair' command. I didn't give an example for that in the original post as you should really consult Oracle Support for assistance with that kind of problem.

      Cheers,
      Bane

      Delete
    2. Hi Bane,

      thanks for the kfed repair command :-) Just dug me out of a hole.

      We tried using the kfed merge option based on other postings and in the end ran the repair command. This brought the ASM diskgroup online and I was able to bring up 5 of the 7 development databases. Still working on the other 2.

      Cheers

      Dave - Reading, UK

      Delete
  2. Hi Dave,

    Good to hear you found the repair option useful. Let me know if you get stuck or if you have any questions on this.

    Cheers,
    Bane

    ReplyDelete