Datastore corruption leads to virtual Machine issues and loss of access to backing datastore.
search cancel

Datastore corruption leads to virtual Machine issues and loss of access to backing datastore.

book

Article ID: 403004

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

 

  • Tasks such as Power Off, vMotion, and Storage vMotion remain stuck
  • Loss of access to Content librarys
  • Datastore backing VMs/Content library's potentiality show as invalid or inaccessible
  • The VMs remain powered on but are unresponsive to all user actions and tasks at the vSphere environment.
  • Browsing the VM directories under /vmfs/volume/########## is failing
  • VMs on this datastore have tasks that remain incomplete
  • In the /var/log/VMkernel file, you see entries similar to:

2024-03-25T08:59:33.659Z Wa(180) vmkwarning: cpu37:2098333)WARNING: HBX: 754: '"Datastore_Name"': HB at offset 0 - Volume ########-########-####-############ on naa.################################:1 has been detected corrupted
 may be damaged on disk. Corrupt heartbeat detected:
2024-03-25T08:59:33.659Z Wa(180) vmkwarning: cpu37:2098333)WARNING:   [HB state 0 offset 0 gen 0 stampUS 0 uuid 00000000-00000000-0000-000000000000 jrnl <FB 0> drv 0.0]
2025-06-25T18:59:33.659Z Wa(180) vmkwarning: cpu37:2098333)WARNING: FS3: 636: VMFS volume "Datastore_Name"/########-########-####-############ on naa.################################:1 has been detected corrupted

  • In the /var/log/vobd.log file, you see entries similar to:

2024-03-25T08:51:39.391Z In(14) vobd[2098055]:  [vmfsCorrelator] 8576524005331us: [vob.vmfs.resource.corruptondisk] Volume ########-########-####-############ ("Datastore_Name") might be damaged on the disk. Resource cluster metadata corruption has been detected.
2024-03-25T08:51:39.391Z In(14) vobd[2098055]:  Event rate limit reached. Dropping vprob: esx.problem.vmfs.resource.corruptondisk

 

 

Environment

VMware vSphere ESXi (All Versions)

Resolution

partedUtil getptbl /vmfs/devices/disks/<device ID>

...where <device ID> is the t10.xxx, naa.xxx, or eui.xxx device ID

You should see output similar to the following:

Device:  /vmfs/devices/disks/naa.################################


Partition table:
gpt
XXXX 255 63 YYYYYYYYY
1 2048 ZZZZZZZZZ AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Usable sectors:
34 ZZZZZZZZZ

If not, then the partition table no longer contains a valid VMFS partition and has been deleted, overwritten, or corrupted or LUN may be overwritten by secure wipe utility or other operating system.

  • VOMA check

The vSphere On-disk Metadata Analyzer (VOMA) scans the VMFS volume metadata and highlights any inconsistencies to which you may be required to open a support request.

To perform a VOMA check on a VMFS datastore and send the results to a specific log file, the command syntax is:

voma -m vmfs -f check -a -d /vmfs/devices/disks/naa.#################:1 -s /tmp/analysis.txt


Where: naa.#################:1 is the LUN NAA ID and partition to be checked.

For a VMFS6 multi extent volume, LUN NAA ID represents the head extent. Note the ":1" at the end of device. This is the partition number containing the datastore.

Partition number is optional when the datastore resides on the first partition, i.e., VOMA cannot be run against the device if datastore resides on a partition other than the first partition.

Note:
1. VOMA is not supported on VMFS5 multi extent volume.

2. On running VOMA against a device, where datastore does not reside on the first partition, it produces an error similar to:

Error: Missing LVM Magic. Disk doesn't have a valid LVM Device

Error: Failed to Initialize LVM Metadata

In circumstances where the corruption is irreversible, VMware recommends that you restore from backup or consult with a data recovery organization as VMware does not perform data recovery. For more information, see Data recovery services for data not recoverable by VMware Technical Support.

Before running VOMA you must ensure that:
All virtual machines on the affected datastore are powered off or migrated to another datastore. See Impact/Risks section.
For VMFS-5/VMFS-6 datastores, the datastore is unmounted on all ESXi 6.x/7.x hosts.
If you attempt to run VOMA against a volume that is still being accessed by one or more ESXi hosts, VOMA produces a message similar to:

voma -m vmfs -d /vmfs/devices/disks/naa.#################:1
Checking if device is actively being used by other hosts
Found 1 actively heart beating hosts on device '/vmfs/devices/disks/naa.#################:1'


VOMA command syntax options
Option Description
-m or --module Specify whether the module to run is vmfs or lvm. When vmfs is specified, the lvm check is also performed. The default check is vmfs.
-d or --device Device/disk to be inspected.
-s or --logfile Used to specify the output log file.
-v or --version Display the VOMA version.
-h or --help Displays the VOMA command help message.

 

Powered by Wolken Software