Shared san vmfs datastore overwritten by system boot partitions during installation

Products

• VMware vSphere ESXi

Issue/Introduction

When attempting to manage or access virtual machines on an ESXi host, you may observe the following symptoms:

• Virtual machines fail during vMotion operations and abruptly power off.

• Virtual machines cannot be powered back on, with errors indicating the backing files are missing or inaccessible.

• A shared SAN LUN displays as unmounted or shows 0 bytes in capacity on some ESXi hosts.

• The backing LUN represents multiple partitions (Partitions 1, 5, 6, 7, 8) typical of an ESXi system boot layout, rather than a single dedicated VMFS partition.

• Running the vSphere On-disk Metadata Analyzer (voma) on the device partitions fails with: LVM magic not found at expected Offset

Environment

ESXi: 7.0 U3

Cause

• This issue occurs when a shared SAN storage LUN hosting a dedicated VMFS datastore is accidentally targeted as the boot installation destination during an ESXi hypervisor deployment. The installation process overwrites the VMFS sector header start boundary (sector 2048) with the ESXi GPT partition layout (partitions 1, 5, 6, 7, and 8), destroying the original datastore metadata.

Resolution

1. Run the following command in the ESXi Shell to check the partition table of the problematic LUN:
   

   partedUtil getptbl "/vmfs/devices/disks/naa.<ID>

2. If the LUN represents an ESXi system layout similar to the following, the LUN has been overwritten:
   

   1 64 204863 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
   5 208896 8595455 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
   ...
   8 268437504 8589934558 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

3. Because the beginning sectors (including block 2048) have been physically overwritten, the previous filesystem is unrecoverable.
4. Unpresent/destroy the corrupted LUN on the SAN array, or wipe the disk headers.
5. Provision a new LUN from your SAN array and present it to the ESXi cluster.
6. Format a new VMFS datastore on the newly provisioned LUN.
7. Restore the affected virtual machines from your last known good backups.