The views expressed on this blog are my own and do not necessarily reflect the views of Oracle

September 22, 2011

amdu - ASM Metadata Dump Utility


The ASM Metadata Dump Utility - better knows as amdu - is used by Oracle Support and Oracle Development to diagnose and sometimes resolve ASM issues. It can be used to print ASM metadata and extract both ASM metadata and database datafiles from ASM disk groups.

The amdu does not depend on the mount state of an ASM instance or the mount state of a disk group, so it can be used with ASM instance down and with dismounted disk groups. It can even be used with damaged or missing ASM disks!

Use amdu to extract a controlfile from a mounted disk group

In the first example I will work with a disk group that is still mounted and will extract one of the controlfiles for database BR.

Let's first locate all controlfiles:

$ asmcmd find --type controlfile + "*"
+DATA/ASM/CONTROLFILE/cfcopy.268.734178331
+DATA/BR/CONTROLFILE/Current.276.723906721
+DATA/BR/CONTROLFILE/Current.285.723908117
+DATA/BR/CONTROLFILE/Current.294.723912823
+DATA/ORCL/CONTROLFILE/Current.260.715782325
+DATA/cfcopy
+RECO/BR/CONTROLFILE/Current.256.723906723
+RECO/BR/CONTROLFILE/Current.260.723908117
+RECO/BR/CONTROLFILE/Current.264.723912823

So we have three copies of the controlfile in disk group DATA and three copies in disk group RECO. I will extract one controlfile, say Current.276.723906721, from disk group DATA.

I need to know the disks for disk group DATA:

$ asmcmd lsdsk -G DATA
Path
ORCL:DISK1
ORCL:DISK2

So disk group DATA has two disks - DISK1 and DISK2, and these are ASMLIB disks (note the prefix 'ORCL'). Strictly speaking I did not need to know the disk names for this particular exercise, as all I need is the ASM discovery string (the value of ASM_DISKSTRING parameter).

Let's extract that controlfile out of the disk group DATA onto the file system:

$ cd /tmp
$ amdu -diskstring="ORCL:*" -extract DATA.276 -output control.276 -noreport -nodir
AMDU-00204: Disk N0001 is in currently mounted diskgroup DATA
AMDU-00201: Disk N0001: 'ORCL:DISK1'
$ ls -l control.276
-rw-r--r-- 1 grid oinstall 9748480 Sep 22 22:42 control.276

The options used were as follows:
-diskstring: either the full path to disk devices or the value of ASM_DISKSTRING parameter.
-extract: the disk group name, full stop, the ASM file number.
-output: the output file name (in the current directory).
-noreport: do not generate the amdu run report
-nodir: do not create dump directory

Use amdu to extract a datafile from a dismounted disk group

Getting the controlfile out of a mounted disk group was fairly straightforward. But back in real life, a customer calls me to see if I can extract that important datafile, from the disk group that doesn't mount, with no backups and they are not sure of the exact name of that datafile. Let's see how I would go about extracting such file.

The objective is to extract a single datafile, named something like NSA, from disk group DATA that cannot be mounted. That means we cannot use sqlplus or asmcmd to locate the datafile.

Let's dump all metadata for disk group DATA

$ cd /tmp
$ amdu -dump DATA -noimage
amdu_2011_09_22_22_57_05/
$ cd amdu_2011_09_22_22_57_05
$ ls -l
total 28
-rw-r--r-- 1 grid oinstall  5600 Sep 22 22:57 DATA.map
-rw-r--r-- 1 grid oinstall 10462 Sep 22 22:57 report.txt

This time the dump directory was created and two files were generated.

File report.txt contains the information about the server, amdu command and options used, disks that are possibly members of the disk group DATA and the information about allocation units (AUs) on those disks. Let's review the contents of the report file:

$ more report.txt
-*-amdu-*-
******************************* AMDU Settings ********************************
ORACLE_HOME = /u01/app/11.2.0/grid
System name:    Linux
Node name:      
Release:        2.6.18-128.4.1.0.1.el5
Version:        #1 SMP Tue Aug 4 15:10:25 EDT 2009
Machine:        i686
amdu run:      
Endianess:      1
...
----------------------------- DISK REPORT N0001 ------------------------------
                Disk Path: ORCL:DISK1
           Unique Disk ID:
               Disk Label: DISK1
     Physical Sector Size: 512 bytes
                Disk Size: 4886 megabytes
               Group Name: DATA
                Disk Name: DISK1
       Failure Group Name: DISK1
              Disk Number: 0
            Header Status: 3
       Disk Creation Time: 2010/03/01 15:07:47.135000
          Last Mount Time: 2011/09/02 15:35:52.676000
    Compatibility Version: 0x0b200000(11020000)
         Disk Sector Size: 512 bytes
         Disk size in AUs: 4886 AUs
         Group Redundancy: 1
      Metadata Block Size: 4096 bytes
                  AU Size: 1048576 bytes
                   Stride: 113792 AUs
      Group Creation Time: 2010/03/01 15:07:46.819000
  File 1 Block 1 location: AU 2
              OCR Present: NO
...
************************** SCANNING DISKGROUP DATA ***************************
            Creation Time: 2010/03/01 15:07:46.819000
         Disks Discovered: 2
               Redundancy: 1
                  AU Size: 1048576 bytes
      Metadata Block Size: 4096 bytes
     Physical Sector Size: 512 bytes
          Metadata Stride: 113792 AU
   Duplicate Disk Numbers: 0
---------------------------- SCANNING DISK N0001 -----------------------------
Disk N0001: 'ORCL:DISK1'
           Allocated AU's: 2563
                Free AU's: 2323
       AU's read for dump: 34
       Block images saved: 6661
        Map lines written: 34
          Heartbeats seen: 0
  Corrupt metadata blocks: 0
        Corrupt AT blocks: 0
...

File DATA.map contains the data map. While much more useful for our purposes, it is also very cryptic. Let's have a look:

$ more DATA.map
N0001 D0000 R00 A00000000 F00000000 I0 E00000000 U00 C00256 S0000 B0000000000
N0001 D0000 R00 A00000001 F00000000 I0 E00000000 U00 C00256 S0000 B0000000000
N0001 D0000 R00 A00000002 F00000001 I0 E00000000 U00 C00256 S0000 B0000000000
N0001 D0000 R00 A00000003 F00000003 I0 E00000001 U00 C00256 S0000 B0000000000
N0001 D0000 R00 A00000004 F00000003 I0 E00000011 U00 C00256 S0000 B0000000000
...
N0001 D0000 R00 A00000234 F00000267 I1 E00000000 U00 C00001 S0000 B0000000000
...
N0004 D0001 R00 A00001512 F00000292 I1 E00000000 U00 C00001 S0000 B0000000000
N0004 D0001 R00 A00002304 F00000290 I1 E00000000 U00 C00003 S0000 B0000000000
N0004 D0001 R00 A00002643 F00000264 I1 E00000000 U00 C00001 S0000 B0000000000

Of the immediate interest are fields A and F. A00000234, for example, is telling me that this line is for AU 234. F00000267 is telling me that this line is about ASM file 267. This will come in handy later on.

Now we need to hunt down that NSA datafile...

ASM metadata file 6 is the alias directory, so that is the first place I will look at. From DATA.map, I can work out AUs for ASM file 6:

$ grep F00000006 DATA.map
N0004 D0001 R00 A00000008 F00000006 I0 E00000000 U00 C00256 S0000 B0000000000

Pure luck - a single line! That tells me two things - there are not many files in this disk group (all their aliases fit in a single AU) and that the alias directory is in allocation unit 8 (A00000008) on disk 1 (D0001). From report.txt, I know that disk 1 is ORCL:DISK2 and that the AU size is 1MB. Let's use kfed to look at the alias directory.

$ ls -l /dev/oracleasm/disks/DISK2
brw-rw---- 1 grid asmadmin 8, 6 Aug 24 14:38 /dev/oracleasm/disks/DISK2

$ kfed read /dev/oracleasm/disks/DISK2 aun=8 | more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
...

Yep, this is indeed the alias directory. Now look for a datafile named NSA:

for (( i=0; i<256; i++ ))
do
  kfed read /dev/oracleasm/disks/DISK2 aun=8 blkn=$i | grep -1 NSA
done

That gave me the following output:

kfade[15].entry.refer.incarn:         0 ; 0x4a4: A=0 NUMM=0x0
kfade[15].name:             NSA_TN_DATA ; 0x4a8: length=11
kfade[15].fnum:                     267 ; 0x4d8: 0x0000010b

So the actual datafile name is NSA_TN_DATA and I can see it is ASM file number 267. With that information I can extract the datafile:

$ amdu -diskstring="ORCL:*" -extract DATA.267 -output NSA_TN_DATA.267 -noreport -nodir

$ ls -l
total 102544
-rw-r--r-- 1 grid oinstall      5600 Sep 22 22:57 DATA.map
-rw-r--r-- 1 grid oinstall 104865792 Sep 22 23:42 NSA_TN_DATA.267
-rw-r--r-- 1 grid oinstall     10462 Sep 22 22:57 report.txt

So what can I do with this file? Well, if I can extract the database controlfile, system and sysaux files, I might be able to use them to open the database. I may be able to plug this file into another database. Maybe I will need to use DUL to extract data from that file...

It is important to note that while the amdu will extract the file - the file itself may be corrupt or damaged in some way. After all there is a reason for the disk group not mounting - chances are the ASM metadata is corrupt or missing, but that can be the case with the datafile data as well. The point it that there is no substitute for backup, so keep that in mind.

The amdu dump triggered on an error

Since ASM version 11.2.0.3, an amdu dump may be triggered automatically by ORA-600 [kfd...] class of errors. When that happens, in addition to the error being logged in the ASM alert log, there will a message indicating an amdu dump as well. The dump will go into the diagnostic dump directory.

Conclusion

The amdu is a very handy utility, but it may be of limited value to an end user or a DBA. Still, knowing about amdu can be useful when dealing with Oracle Support.

4 comments:

  1. Can you run AMDU periodically to detect ASM metadata corruptions ? All Oracle documents on detecting and avoiding block corruptions do not mention ASM metadata corruptions at all.

    ReplyDelete
  2. The amdu is not a tool for that. You can extract both data and metadata with amdu, but it neither detects not prevents the metadata corruption.

    The ASM command 'ALTER DISKGROUP ... CHECK' will detect and if possible repair the metadata corruptions. That command cross checks the ASM metadata structures and if the corruption is not significant it may be able to repair the damage.

    The 'kfed repair' will correct ASM disk header corruptions (11.1.0.7 and later).

    So there are options to do what you want. Yes, you are correct about these not being documented as both are undocumented utilities.

    Finally, there is no substitute for backups (data backups that is). If you have all you data backed up, disk group corruption is not such a scary prospect.

    Cheers,
    Bane

    ReplyDelete
  3. excellent demo ! ... good to know of such possibilities with ASM.

    ReplyDelete