VM operations fail due to VMFS overwrite by another guestOS

Products

• VMware vSphere ESXi

Issue/Introduction

VM operations may fail, vMotion operations may fail to complete, or the datastore may appear inaccessible in the vSphere Client. This issue occurs if a LUN was re-initialized by another OS (like Windows or Linux) or if a storage controller mistakenly wrote a different partition over an existing VMFS volume.

• At least some VM power operations fail.
  
  
• vMotion operations to/from the datastore fail to complete.
  
  
• The datastore is shown as inaccessible in the vSphere Client.
  
  
• The LUN backing the datastore is healthy at the storage array level.

Environment

VMware vSphere ESXi (all versions)

Cause

A LUN has been formatted or "initialized" by an external process or utility, such as a Windows OS installation. This leads to the overwrite of:

• The device partition table.
• Key VMFS metadata (the LVM header at offset 0x200000 and VMFS header at offset 0x1400000).

Evidence of this overwrite often appears as:

• FAT16 or Windows partition signatures within the VMFS metadata space.
• Log errors such as FS3DescriptorVerify failed and Corrupt heartbeat detected.

Example:

• Hexdump from a healthy VMFS volume:
  
  Note:
  Partition table (UEFI PART)
  LVM begns at offset 0020000
  VMFS filesystem begins at offset 01400000
  
  
  
  
  
  
• Hexdump of a LUN where VMFS metadata is overwritten by another guestOS
  
  hexdump -C /vmfs/devices/disks/naa.################################ | less
  
  
  
  

• • The hexdump shows evidence of X.MSDOS5.0 and FAT16 filesystem overwrites of the VMFS metadata. VMFS does not use FAT16.  The presence of a FAT16 file system header within the same space where VMFS metadata is expected confirms that an external process or utility (likely a non-ESXi OS) attempted to format or "initialize" this LUN.
    
    
  • The hexdump output clearly shows that LVM header (at offset 00200000) and VMFS header (at offset 01400000) contain overwritten data, which happens due to external factors.
    
    
  • Data is written at offsets which zeroed in VMFS volume metadata.
    
    
     
• The device partition table has been overwritten:
  
  partedUtil getptbl /vmfs/devices/disks/naa.################################

         For example, the following partedUtil getptbl ouput shows that the device has been overwritten with a Windows partition table.
         Note: partition 1 with guid DE94BBA406D14D40A16ABFD50179D6AC indicates a Windows recovery partition.

         

 

• Because the VMFS metadata has been compromized, /var/run/log/vobd.log reports datastore corruption: 

  [vmfsCorrelator] [vob.vmfs.resource.corruptondisk] Volume ###-###-###-###(<VMFS_Datastore>) might be damaged on the disk. Resource cluster metadata corruption has been detected corrupted.

• voma check reports "LVM magic not found at expected Offset"
  
  voma -m vmfs -f check -d /vmfs/devices/disks/naa.################################
  
  Running VMFS Checker version 2.1 in check mode
  Initializing LVM metadata, Basic Checks will be done
  Initializing LVM metadata..-
  LVM magic not found at expected Offset,
  It might take long time to search in rest of the disk.
  Do you want to continue (Y/N)?

  0) _Yes
  1) _No

Resolution

1. Analyze Log Files: Review /var/run/log/vmkernel.log for corruption signatures.
   • Look for FS3DescriptorVerify failed and Corrupt heartbeat detected.
2. Verify LUN Integrity (Hexdump): Use the following command to check for non-VMFS headers: hexdump -C /vmfs/devices/disks/naa.#### | less
   • Note: If FAT16 or other OS signatures are present in the VMFS header space, the volume has been overwritten.
3. Determine Recovery Path:
   1. If LVM/VMFS headers are overwritten: The filesystem is destroyed. Do not attempt VOMA repair as it will not work. Proceed to restore from a valid backup.
   2. If headers are intact but structural errors exist: Run VOMA to validate if the corruption is fixable.
4. Restore: If recovery is not possible, restore the affected virtual machines from a valid backup.
5. Data Recovery: For data not recoverable by VMware Technical Support, engage Data recovery services.

Additional Information

Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency

Capture VOMA Dump for VMFS Datastore

Checking Metadata Consistency with vSphere On-disk Metadata Analyzer

OSDATA volume has been detected corrupted