Inaccessible VMs and Datastore Issues Due to Metadata Corruption
search cancel

Inaccessible VMs and Datastore Issues Due to Metadata Corruption

book

Article ID: 391481

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware Telco Cloud Platform VMware Cloud Foundation

Issue/Introduction

  • VMs on ESXi host may show invalid/inaccessible.

  • Unable to perform VM's storage Migration

  • Datastores do not show up

  • Multiple datastores reported that the LUN has been detected corrupt.

  • The issue is seen when there is corruption on VMFS datastore.

  • On ESXi host /var/run/log/vobd.log show the below corrupt events for the datastore.

    2025-03-17T19:47:33.594Z cpu7:29385420)WARNING: HBX: 751: ' LUN-xxxx: HB at offset - Volume 5#####-1f###fc-###-####### may be damaged on disk. Corrupt heartbeat detected:
    2025-03-17T19:47:33.594Z cpu7:29385420)WARNING: FS3: 608: VMFS volume LUN-###/5f####7-1f###c-###-####### on eui.2d781#########99########:1 has been detected corrupted

  • The /var/run/log/vmkernel.log file contains these events:

    2025-03-18T01:19:29.973Z: Event rate limit reached. Dropping vprob: esx.problem.vmfs.heartbeat.corruptondisk
    2025-03-18T01:19:29.984Z: [vmfsCorrelator] 8411869508265us: [vob.vmfs.heartbeat.corruptondisk] Volume 5####8d-e2###dc-2##6-####### (" LUN-####") may be damaged on disk. Corrupt heart beat detected at offset 41###6

  • When verifying the partition table for the LUN associated with the datastore, the below error would show up.

    [root@<Host-Name>:partedUtil getptbl /vmfs/devices/disks/eui.90e9dff##################
Error: The primary GPT table is corrupt, but the backup appears OK, so that will be used. Fix primary table? diskPath (/dev/disks/eui.90e9dff################) diskSize (10485760000) AlternateLBA (1) LastUsa bleLBA (10485759966)
gpt
652708 255 63 10485760000
1 2048 10485759966 AA31E0######B9590######1D1B8 vmfs


  • When running VOMA on the LUN/Device, VOMA gives the error:

    [root@<Host-Name>: voma -m vmfs -f check -d /vmfs/devices/disks/eui.90e9####26c1###c9ce9######## Running VMFS Checker version 2.1 in check mode
Initializing LVM metadata, Basic Checks will be done
Initializing LVM metadata..-
LVM magic not found at expected Offset,

Environment

ESXi: 8.x, 7.0

VCF: 5.x, 4.x

TCP: 5.x

Cause

External overwriting or severe metadata corruption of the VMFS header/LVM magic region.

Resolution

  • As the LVM magic not found for the LUN, it confirms the corruption on the LUN that cannot be fixed using VOMA.
  • Evaluate LUN recovery options from storage by contacting internal storage team or contacting storage vendor.
  • If there is no LUN level backup, it may require restoring the VMs from backup on to a different datastore.

Additional Information

Powered by Wolken Software