Use VOMA to Check Metadata Consistency

Products

VMware vSphere ESX 6.x VMware vSphere ESX 7.x VMware vSphere ESX 8.x

Issue/Introduction

You can check metadata consistency when you experience problems with a VMFS datastore or a virtual flash resource. For example, perform a metadata check if one of the following occurs:

You experience storage outages.
After you rebuild RAID or perform a disk replacement.
You see metadata errors in the vmkernel.log file similar to the following:

cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore_name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $
You are unable to access files on a VMFS.
You see corruption being reported for a datastore in events tabs of vCenter Server.

Environment

Vsphere ESXi 6.x
Vsphere ESXi 7.x
Vsphere ESXi 8.x

Cause

Check metadata consistency and identify and fix incidents of metadata corruption on the VMFS datastores or a virtual flash resource or logical volumes that back the VMFS datastores.
Identify incidents of metadata corruption that affect file systems or underlying logical volumes.

Resolution

Prerequisites

Power off any virtual machines that are running or migrate them to a different datastore.

Procedure
1. Obtain the name and partition number of the device that backs the VMFS datastore that you want to check.
  - #esxcli storage vmfs extent list
    
    The Device Name and Partition columns in the output identify the device. For example:
    
    Volume Name ..... Device Name Partition
    
    1TB_VMFS6 ..... naa.xxxx 3
2. Check for VMFS errors.
  - Provide the absolute path to the device partition that backs the VMFS datastore, and provide a partition number with the device name. For example:
    # voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x
  - The output lists possible errors. For example, the following output indicates that the heartbeat address is invalid.
    
    XXXXXXXXXXXXXXXXXXXXXXX
    Phase 2: Checking VMFS heartbeat region
    ON-DISK ERROR: Invalid HB address
    Phase 3: Checking all file descriptors.
    Phase 4: Checking pathname and connectivity.
    Phase 5: Checking resource reference counts.
    
    Total Errors Found: 1

Additional Information

VOMA Command Options
Command Option	Description
-m\|--module	The modules to run include the following:
	vmfs	If you do not specify the name of the module, this option is used by default. You can check the VMFS file systems and the file systems that back virtual flash resources. If you specify this module, minimal checks are performed for LVM as well.
	lvm	Check logical volumes that back the VMFS datastores.
	ptbl	Check and validate VMFS partitions, such as MBR or GPT. If no partition exists, determine whether partitions should exist.
-f\|--func	Functions to be performed include the following:
	query	List functions supported by module.
	check	Check for errors.
	fix	Check and fix errors.
	dump	Collect metadata dump.
-a\|--affinityChk	Include affinity-related check and fix for VMFS6.
-d\|--device	Indicate the device or disk to inspect. Make sure to provide the absolute path to the device partition backing the VMFS datastore. If the datastore spans multiple devices, provide the UUID of the head extent. For example, voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x If you use the -x\|--extractDump command, enter multiple device paths, with a partition qualifier, separated with a comma. The number of device paths you enter equals the number of spanned devices.
-b\|--blockSize	Indicate the disk block size.
-s\|--logfile	Specify the path to log file to output the results.
-x\|--extractDump	Extract the collected dump using VOMA.
-D\|--dumpfile	Indicate the dump file to save the collected metadata dump.
-v\|--version	Display the version of VOMA.
-h\|--help	Display the help message for the VOMA command.
-Y	Indicate that you run VOMA without using PE tables for address resolution.
-Z\| --file	Indicate that you run VOMA on extracted device files.

Use VOMA to Check Metadata Consistency

Article ID: 382692

Updated On: 11-23-2024