Corruption seen on VMFS datastore due to overwritten data

Products

• VMware vSphere ESXi

Issue/Introduction

• vMotion operation to/from corrupt datastore remains stuck.
  
  
• Datastore is showing up as inaccessible on vCenter.

Environment

VMware vSphere ESXi 

Cause

• This often happens if a disk was re-initialized by another OS (like Windows or Linux) or if a storage controller mistakenly wrote a different partition style over an existing VMFS volume. In below example, X.MSDOS5.0 and FAT16 filesystem can be seen. VMFS does not use FAT16.  The presence of a FAT16 file system header within the same space where VMFS metadata is expected confirms that an external process or utility (likely a non-ESXi OS) attempted to format or "initialize" this LUN, overwriting the VMFS volume's "Lock" and "Heartbeat" regions.
  
  Example -
  
  The hexdump output clearly shows that LVM header (00200000) and VMFS header (01400000) contain overwritten data, which happens due to external factors. 

hexdump -C /vmfs/devices/disks/naa.################################ | less

• We may see that the device partition table has been overwritten. This can be confirmed by running:
  
  partedUtil getptbl /vmfs/devices/disks/naa.################################

         For example, the following partedUtil getptbl ouput shows that the device has been overwritten with a Windows partition table.
         Note: partition 1 with guid DE94BBA406D14D40A16ABFD50179D6AC indicates a Windows recovery partition.

         

• In /var/run/log/vobd.log of host, we can see below corruption entry:
  
  [vmfsCorrelator] [vob.vmfs.resource.corruptondisk] Volume ###-###-###-###(<VMFS_Datastore>) might be damaged on the disk. Resource cluster metadata corruption has been detected corrupted.



Resolution

• Run VOMA to validate if the corruption issue can be fixed. Note: While running VOMA advance fix, if the corruption is too severe, VOMA can automatically delete the corrupt files.
  
  
• If key filesystem markers are overwritten VOMA will not run.
• Validate the logs again post running VOMA, if the corruption events are still seen then restore the VMs on the corrupt volume from backup or engage Data Recovery Services.
  Data recovery services for data not recoverable by VMware Technical Support
  
  
  
  

Additional Information

Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency

Capture VOMA Dump for VMFS Datastore

Checking Metadata Consistency with vSphere On-disk Metadata Analyzer

OSDATA volume has been detected corrupted