Error: "VMFS datastore reported datastore corrupted" during ESXi upgrade/storage outage

Products

• VMware vSphere ESXi

Issue/Introduction

• During ESXi host upgrade VMs went offline
• There was a storage issue in the environment with a VM with Fault Tolerance enabled.
• The secondary VM got corrupted and Fault Tolerance cannot be enabled again.
• Once Fault Tolerance is enabled for the VM, VM changes state to power off
• 

Environment

VMware vSphere ESXi 8.0.X
VMware vSphere ESXi 7.0.X

Cause

• LUN that was backing the datastore was inadvertently re-used to install ESXi
  
  
• We observe the following in the /var/run/log/vmkernel.log 

  Wa(180) vmkwarning: cpu97:2097913)WARNING: FS3: 636: VMFS volume OSDATA-########-#######-####-###########/684a3c30-########-bdbf-############ on naa.################################:7 has been detected corrupted
  In(182) vmkernel: cpu97:2097913)FS3: 639: While filing a PR, please report the names of all hosts that attach to this LUN, tests that were running on them,
  In(182) vmkernel: cpu97:2097913)FS3: 662: and upload the dump by `voma -m vmfs -f dump -d /vmfs/devices/disks/naa.################################:7 -D X`
  In(182) vmkernel: cpu97:2097913)FS3: 665: where X is the dump file name on a DIFFERENT volume

• cpu32:2128233)WARNING: DLX: 1017: Volume ########-########-####-############ ("###-####-####") might be damaged on the disk. Corrupt lock detected at offset 89849856
  
  cpu32:2128233)WARNING: [type cc7c444e offset 1096249693888711131 v 3165975273340201150, hb offset 1141792724207585790
  
  cpu32:2128233)WARNING: FS3: 608: VMFS volume ###-####-####/########-########-####-############ on naa.################################:1 has been detected corrupted

Resolution

1. This change is "irreversible" hence, the virtual machines cannot be recovered as the physical blocks on LUN are overwritten.
2. Clone the datastore OR
3. Restore from valid backups OR
4. Rebuild the virtual machines