Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency

Products

VMware vSphere ESXi

Issue/Introduction

This article provides instructions on performing a file system metadata check using the vSphere On-disk Metadata Analyzer (VOMA).

For large-scale environments, such as VDI deployments, 'Invalid metadata' errors are often linked to transient storage connectivity issues rather than physical metadata corruption. Before proceeding with VOMA, verify the health of the storage fabric, switches, and pathing to rule out transient I/O inconsistencies.

VOMA can be used to check the following:

Partition table update
Reports of metadata errors in the vmkernel.log file
Unable to access files on the VMFS volume that are not in use by any other host

Run VOMA if you experience any of the following:

You have problems accessing certain files on a VMFS datastore.
You cannot modify or erase files on a VMFS datastore.
Attempting to read files on a VMFS datastore may fail with the error:

Invalid argument
Attempting to modify the partition table fails with the error:

Error: Read-only file system during write on /dev/disks/naa.xxx
SetPtableGpt: Unable to commit to disk
ESXi - /var/log/vmkernel.log file

vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 130: <START vm.vswp>
vmkernel: 25:21:39:57.861 cpu15:1047)Lock [type 10c00001 offset 52076544 v 69, hb offset 4017152
vmkernel: gen 109, mode 1, owner ########-#########-####-############ mtime 3420]
vmkernel: 25:21:39:57.861 cpu15:1047)Addr <4, 1011, 10>, gen 36, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
vmkernel: 25:21:39:57.861 cpu15:1047)len 3221225472, nb 3072 tbz 0, zla 3, bs 1048576
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 132: <END vm.vswp>
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/########-#########-####-############/vm/vm.vswp': Invalid metadata
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 3586: Failed to initialize swap file '/volumes/########-#########-####-############/vm/vm.vswp': Invalid metadata
cpu11:268057)WARNING: HBX: 599: Volume ########-#########-####-############ ("<Datastore name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Resolution

The vSphere On-disk Metadata Analyzer (VOMA) scans the VMFS volume metadata and highlights inconsistencies. Note: VOMA only checks for metadata corruption; it does not check for underlying storage or disk physical errors.

Prerequisites & Safety Warnings

All virtual machines on the affected datastore must be powered off or migrated to another datastore.
For VMFS-5/VMFS-6, the datastore must be unmounted on all ESXi 7.x, 8.x, and 9.x hosts. (Note: When attempting to run VOMA against a volume that is still being accessed, VOMA will fail and report the MAC address of the actively heartbeating host).

voma -m vmfs -d /vmfs/devices/disks/naa.#################:1
Checking if device is actively being used by other hosts
Found 1 actively heart beating hosts on device '/vmfs/devices/disks/naa.#################:1'
1): MAC Address ##:##:##:##:##:##

Impact/Risks

Shutting down a virtual machine running on files having certain types of corrupt metadata by the vSphere On-disk Metadata Analyzer, may make the virtual machine and its data permanently unavailable.
Because of this, it is always advisable to have current backups of the virtual machines in the environment. If you suspect that the virtual machine may become unavailable (for example, read/write errors in the guest OS, or the virtual machine is unresponsive), open a support request.

Procedure

SSH to an ESXi host via root
Identify the LUN NAA ID and partition number of the affected datastore by running:

esxcli storage vmfs extent list
Execute the VOMA check and output the results to a log file. Use the following syntax, replacing the NAA ID with your device:

voma -m vmfs -f check -a -d /vmfs/devices/disks/naa.#################:1 -s /tmp/analysis.txt

Note: For a VMFS-6 multi-extent volume, the LUN NAA ID represents the head extent. The :1 indicates the partition number. This is the partition number containing the datastore. Partition number is optional when the datastore resides on the first partition, i.e., VOMA cannot be run against the device if the datastore resides on a partition other than the first partition.

Notes

VOMA is not supported on VMFS-5 multi extent volumes.
Running VOMA against a device where the datastore does not reside on the first partition, produces an error similar to:

Error: Missing LVM Magic. Disk doesn't have a valid LVM Device
Error: Failed to Initialize LVM Metadata

VOMA Command Syntax Options

Option	Description
`-m` or `--module`	Specify the module: vmfs or lvm. When vmfs is specified, the lvm check is also performed. (Default is vmfs).
`-f` or `--func`	Specify the function to perform, such as check.
`-d` or `--device`	Device/disk to be inspected (the LUN NAA ID and partition).
`-s` or `--logfile`	Used to specify the output log file.
`-v` or `--version`	Display the VOMA version.
`-h` or `--help`	Displays the VOMA command help message.

If VOMA detects corruption, engage Broadcom Support to attempt a repair. In circumstances where the metadata corruption is irreversible, VMware recommends restoring from a backup or consulting with a data recovery organization as VMware does not perform data recovery. For more information, see Data recovery services for data not recoverable by VMware Technical Support.

Additional Information

Unable to access certain files on a VMFS datastore
Data recovery services for data not recoverable by VMware Technical Support
Checking Metadata Consistency with vSphere On-disk Metadata Analyzer