Viewing and interpreting VMFS metadata

Products

• VMware vSphere ESXi

Issue/Introduction

In some cases it may be necessary to investigate vm files for locked files. Failed backup jobs are often the first indicator that there may be lock contention on a vm disk. 

Environment

The method in this article assumes that the vm is residing on some kind of VMFS backed storage (Fibre or iSCSI SAN, in most cases). 

Cause

VMFS lock contention is the most common cause of snapshot related failures on vm disks.

Resolution

In some cases, when vCenter is unavailable, for example, it may be necessary to manually examine the VMFS lock state. On VMFS backed storage you can simply dump the VMFS metadata.

vmkfstools -D /vmfs/volumes/<datastore_name>/<vm_name>/<vm_file_name>

Alternatively run from within the vm folder:

vmkfstools -D <vm_file_name>

One of the first things to look for is, is the file locked? 

12:16:02:04.331 cpu2:4226)FS3: 142: <START myTestVM-flat.vmdk>
12:16:02:04.331 cpu2:4226)Lock [type 10c00001 offset 42491904 v 588, hb offset 3579904
gen 7, mode 1, owner 00000000-00000000-0000-000000000000 mtime 5312]
12:16:02:04.331 cpu2:4226)Addr <4, 87, 196>, gen 559, links 1, type reg, flags 0x0, uid 0, gid 0, mode 100600
12:16:02:04.331 cpu2:4226)len 8388608, nb 8 tbz 8, cow 0, zla 1, bs 1048576
12:16:02:04.331 cpu2:4226)FS3: 144: <END myTestVM-flat.vmdk>

The example above shows a locked disk, noted by mode:1. An unlocked disk is indicated by mode:0.  The string 000000000000 would normally indicate the MAC address of the owning NIC. In cases where you see this field is all 0s:

# vmkfstools -D <some_file>

A similar output is seen: 

12:16:02:04.331 cpu2:4226)FS3: 142: <START myTestVM-flat.vmdk>
12:16:02:04.331 cpu2:4226)Lock [type 10c00001 offset 42491904 v 588, hb offset 3579904
gen 7, mode 1, owner 00000000-00000000-0000-000000000000 mtime 5312]
12:16:02:04.331 cpu2:4226)Addr <4, 87, 196>, gen 559, links 1, type reg, flags 0x0, uid 0, gid 0, mode 100600
12:16:02:04.331 cpu2:4226)len 8388608, nb 8 tbz 8, cow 0, zla 1, bs 1048576
12:16:02:04.331 cpu2:4226)FS3: 144: <END myTestVM-flat.vmdk>

 

Use the HB offset from the output (highlighted in red) and run this command to get the MAC address of the ESX/ESXi host with the lock:

# hexdump -C /vmfs/volumes/<datastore>/.vh.sf -n 512 -s <offset>

In this example, <offset> is equal to 3579904.

Similar output is seen:

0036a000 02 ef cd ab 00 a0 36 00 00 00 00 00 07 00 00 00 |......6.........|
0036a010 00 00 00 00 d9 a4 d7 d7 fe 00 00 00 80 9d ac 4c |...............L|
0036a020 6d b0 4d 37 1d 8e 02 1a 64 e6 22 67 c1 23 59 00 |m.M7....d."g.#Y.|
0036a030 07 00 00 00 21 00 00 00 00 00 00 00 00 00 00 00 |....!...........|
0036a040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

In this example, the 7th to 12th byte contain the MAC address, 02 1a 64 e6 22 67.

Note: An HB record is not a lock by itself. It only protects the locks.

 

To find the host with the lock, run any one of these commands on each host:

• In ESX/ESXi, esxcfg-nics -l

This command lists all the NICs and the MAC address.

In ESX hosts – # ifconfig | grep HW

There will be an ouptut similar to this:

vmnic0 Link encap:Ethernet HWaddr 00:1A:64:E5:28:64
vmnic1 Link encap:Ethernet HWaddr 00:1A:64:E5:28:66
vmnic2 Link encap:Ethernet HWaddr 00:15:17:B0:73:9E
vmnic3 Link encap:Ethernet HWaddr 00:15:17:B0:73:9F
vswif0 Link encap:Ethernet HWaddr 00:50:56:49:8F:89
vusb0 Link encap:Ethernet HWaddr 02:1A:64:E6:22:67 <-------- MAC address in HB record

In ESXi hosts – # esxcfg-info -n | grep -B 4 "MAC Address"

Note: Alternatively, navigate to Configuration > Network Adapters in the vSphere Client.

Typically, there is a vmnic seen. However, it can be any device with a MAC address (other than a generated MAC address for vswif) on the ESX or ESXi host.

 

 

 In cases where the MAC address belongs to a host that the vm is not registered to, you may be able to clear the lock by placing that host into maintenance mode, and retesting. If in the post test the MAC changes, this is likely held by a backup proxy vm, see the example below. If the MAC does not change, reboot the owning host to clear the lock.

 

A common scenario is a read only lock. This is typically the result of an incomplete snapshot based backup task.

 

# vmkfstools -D myTestVM-flat.vmdk

12:16:02:04.331 cpu2:4226)FS3: 142: <START myTestVM-flat.vmdk>
12:16:02:04.331 cpu2:4226)Lock [type 10c00001 offset 42491904 v 588, hb offset 3579904
gen 7, mode 2, owner 00000000-00000000-0000-000000000000 mtime 5312]
12:16:02:04.331 cpu2:4226)Addr <4, 87, 196>, gen 559, links 1, type reg, flags 0x0, uid 0, gid 0, mode 100600
12:16:02:04.331 cpu2:4226)len 8388608, nb 8 tbz 8, cow 0, zla 1, bs 1048576
12:16:02:04.331 cpu2:4226)FS3: 144: <END myTestVM-flat.vmdk>

In the example above, you can see that the mode is 2, for a read-only lock. You can use the owner MAC or if provided the RO lock owner's MAC. In many cases you will want to check for backup proxy vms being resident on the owning host of the implicated MAC address. It is not unusual for backup proxies to have vm disks attached when a backup job does not terminate or clean up properly after a backup job attempt. You can manually inspect the virtual disks attached and look for disks that do not natively belong to the backup proxy itself. Detaching these disks should clear those locks.

 

 

Additional Information

A summary of lock states based on mode.

 

mode 0 = no lock
mode 1 = exclusive lock (typical for base disks and *.vswp files of a powered on VM)
mode 2 = read only lock (typical for base disks when the VM is running on a snapshot)
mode 3 = multi-writer lock (typical disks in MSCS or FT VMs)