VMs show inaccessible and shared datastores missing from ESXi hosts.
search cancel

VMs show inaccessible and shared datastores missing from ESXi hosts.

book

Article ID: 391481

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware Telco Cloud Platform VMware Cloud Foundation

Issue/Introduction

  • VMs on ESXi host may show invalid/inaccessible.

  • Unable to perform VM's storage Migration

  • Datastores does not show up on ESXi host.

  • Multiple datastores reported that the LUN has been detected corrupt.

  • The issue is seen when there is corruption on VMFS datastore.
  • On ESXi host /var/run/log/vobd.log show the below corrupt events for the datastore.

    2025-03-17T19:47:33.594Z cpu7:29385420)WARNING: HBX: 751: ' LUN-xxxx: HB at offset - Volume 5###xx-1f###fc-###-9######96 may be damaged on disk. Corrupt heartbeat detected:
    2025-03-17T19:47:33.594Z cpu7:29385420)WARNING: FS3: 608: VMFS volume LUN-###/5f###x7-1f###c-###-98######96 on eui.2d781#########99###xxdc5b:1 has been detected corrupted
  • Also, /var/run/log/vmkernel.log would report the below events.
    2025-03-18T01:19:29.973Z: Event rate limit reached. Dropping vprob: esx.problem.vmfs.heartbeat.corruptondisk
    2025-03-18T01:19:29.984Z: [vmfsCorrelator] 8411869508265us: [vob.vmfs.heartbeat.corruptondisk] Volume 5###X8d-e2###dc-2XX6-94###Xbf40 (" LUN-###X") may be damaged on disk. Corrupt heart beat detected at offset 41###6
  • When verifying the partition table for the LUN associated with the datastore, the below error would show up.
    [root@<Host-Name>:/vmfs/volumes/5f###ec-ca###x8c-###-94###xxbee0/log] partedUtil getptbl /vmfs/devices/disks/eui.90e9dff#########xxx0997fdc5b
    Error: The primary GPT table is corrupt, but the backup appears OK, so that will be used. Fix primary table? diskPath (/dev/disks/eui.90e9dff#########xxfdc5b) diskSize (10485760000) AlternateLBA (1) LastUsa bleLBA (10485759966)
    gpt
    652708 255 63 10485760000
    1 2048 10485759966 AA31E0###xxxB9590######1D1B8 vmfs 
  • When running VOMA on the LUN/Device, VOMA gives the error:
    [root@<Host-Name>:/vmfs/volumes/5ff6eaec-c###x8c-###-94f###xxxee0/log] voma -m vmfs -f check -d /vmfs/devices/disks/eui.90e9###x26c1###c9ce9###xfdc5b Running VMFS Checker version 2.1 in check mode
    Initializing LVM metadata, Basic Checks will be done
    Initializing LVM metadata..-
    LVM magic not found at expected Offset,
    It might take long time to search in rest of the disk.
    VMware ESXi Question:
    Do you want to continue (Y/N)?
    0) Yes
    1) No
    Select a number from 0-1: 0
    Searching the device for LVM Magic..

Environment

ESXi: 8.x, 7.0

VCF: 5.x, 4.x

TCP: 5.x

Cause

 

Resolution

  • As the LVM magic not found for the LUN, it confirms the corruption on the LUN that cannot be fixed using VOMA.
  • Check for the LUN recovery options from storage by contacting internal storage team or contacting storage vendor.
  • If there is no LUN level backup, it may require restoring the VMs from backup on to a different datastore.

Additional Information

Powered by Wolken Software