VMs show inaccessible and shared datastores missing from ESXi hosts.
search cancel

VMs show inaccessible and shared datastores missing from ESXi hosts.

book

Article ID: 391481

calendar_today

Updated On: 07-16-2025

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • VMs on ESXi host may show invalid/inaccessible.

  • Datastores does not show up on ESXi host.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

  • Multiple datastores reported that the LUN has been detected corrupt.

  • The issue is seen when there is corruption on VMFS datastore.

On ESXi host /var/run/log/vobd.log show the below corrupt events for the datastore.

2025-03-17T19:47:33.5882 cpu39:29386149) WARNING: FS3: 608: VMFS volume LUN-###/61###x-38###4c-###-94###xxxd78 on eui.352699######c9ce900###xc5b:1 has been detected corrupted
2025-03-17T19:47:33.594Z cpu7:29385420)WARNING: HBX: 751: ' LUN-xxxx: HB at offset - Volume 5###xx-1f###fc-###-9######96 may be damaged on disk. Corrupt heartbeat detected:
2025-03-17T19:47:33.594Z cpu7:29385420)WARNING: FS3: 608: VMFS volume LUN-###/5f###x7-1f###c-###-98######96 on eui.2d781#########99###xxdc5b:1 has been detected corrupted

Also, /var/run/log/vmkernel.log would report the below events.

2025-03-18T01:19:29.973Z: Event rate limit reached. Dropping vprob: esx.problem.vmfs.heartbeat.corruptondisk
2025-03-18T01:19:29.984Z: [vmfsCorrelator] 8411869508265us: [vob.vmfs.heartbeat.corruptondisk] Volume 5###X8d-e2###dc-2XX6-94###Xbf40 (" LUN-###X") may be damaged on disk. Corrupt heart beat detected at offset 41###6

  • When verifying the partition table for the LUN associated with the datastore, the below error would show up.

Error: The primary GPT table is corrupt, but the backup appears OK, so that will be used. Fix primary table?

Example as below:

[root@esx03:/vmfs/volumes/5f###ec-ca###x8c-###-94###xxbee0/log] partedUtil getptbl /vmfs/devices/disks/eui.90e9dff#########xxx0997fdc5b
Error: The primary GPT table is corrupt, but the backup appears OK, so that will be used. Fix primary table? diskPath (/dev/disks/eui.90e9dff#########xxfdc5b) diskSize (10485760000) AlternateLBA (1) LastUsa bleLBA (10485759966)
gpt
652708 255 63 10485760000
1 2048 10485759966 AA31E0###xxxB9590######1D1B8 vmfs 0

  • When running VOMA on the LUN/Device, VOMA gives the error "LVM magic not found at expected Offset,".

    Example below:
     
    [root@esxx1:/vmfs/volumes/5ff6eaec-c###x8c-###-94f###xxxee0/log] voma -m vmfs -f check -d /vmfs/devices/disks/eui.90e9###x26c1###c9ce9###xfdc5b Running VMFS Checker version 2.1 in check mode
    Initializing LVM metadata, Basic Checks will be done
    Initializing LVM metadata..-
    LVM magic not found at expected Offset,
    It might take long time to search in rest of the disk.
    VMware ESXi Question:
    Do you want to continue (Y/N)?
    0) Yes
    1) No
    Select a number from 0-1: 0
    Searching the device for LVM Magic..

Resolution

As the LVM magic not found for the LUN, it confirms the corruption on the LUN that cannot be fixed using VOMA.

Check for the LUN recovery options from storage by contacting internal storage team or contacting storage vendor.

If there is no LUN level backup, it may require restoring the VMs from backup on to a different datastore.

Additional Information

Specialist data recovery providers may be able to reconstruct overwritten metadata and recover access to a file, if the relevant metadata but not the file data has been overwritten. ref: Data recovery services for data not recoverable by VMware Technical Support

Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency