At least one corrupt on-disk heartbeat region was detected on volume causing inaccessible VMs

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms: 

• VMs are inaccessible
  
  
• /var/run/log/vmkernel.log shows
  
  At least one corrupt on-disk heartbeat region was detected on volume ##########-##########-92b0-##########(P##_#######_###_PH). Other regions of the volume might be damaged too.


• partedUtil output says the Primary GPT table is corrupted 

  [root@esxi:~] partedUtil getptbl /vmfs/devices/disks/naa.60000######################13937
Error: The primary GPT table is corrupt, but the backup appears OK, so that will be used. Fix primary table ? diskPath (/dev/disks/naa.60000######################13937) diskSize (42949674240) AlternateLBA (1) LastUsableLBA (42949674206)
gpt
2673493 255 63 42949674240
1 2048 42949672959 AA31E02A400F11DB9590000C2911D1B8 vmfs 0 
  
  
•  hexdump -C /vmfs/devices/disks/naa.60000######################13937 | less shows repeating pattern 
  
  00000000  89 89 89 89 89 89 89 89  89 89 89 89 89 89 89 89  |................|
*
0028b200  f7 f7 f7 f7 f7 f7 f7 f7  f7 f7 f7 f7 f7 f7 f7 f7  |................|
*
003da400  7c 7c 7c 7c 7c 7c 7c 7c  7c 7c 7c 7c 7c 7c 7c 7c  ||||||||||||||||||
*
004c1000  89 89 89 89 89 89 89 89  89 89 89 89 89 89 89 89  |................|
*
006b8400  f7 f7 f7 f7 f7 f7 f7 f7  f7 f7 f7 f7 f7 f7 f7 f7  |................|
*
008fb400  89 89 89 89 89 89 89 89  89 89 89 89 89 89 89 89  |................|
*
009a8400  7c 7c 7c 7c 7c 7c 7c 7c  7c 7c 7c 7c 7c 7c 7c 7c  ||||||||||||||||||
*
  
  

Environment

• vSphere ESX 9.x
• vSphere ESXi 8.x
• vSphere ESXi 7.x

Cause

The HPE ESXi host was decommissioned. However, because the LUN was not unpresented or detached from the host prior to decommissioning, the subsequent Secure Delete initiated on the host filesystems inadvertently overwrote the attached datastores.

Resolution

Decommissioning Procedure Checklist

During ESXi Host Decommissioning, ensure the following steps are completed:

• Unmount the shared datastores.

• Detach the associated LUNs from the host. 
  ref: Detach a LUN device from ESXi host

• Un-present the LUNs from the storage array.