Irrecoverable data loss after deleting a VMFS datastore and creating a new volume

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

When a VMFS datastore is removed from a vSphere environment using the "Delete" operation, the action is destructive, irreversible and immediate. It bypasses the standard "Signature Conflict" safety checks often seen with snapshot LUNs.
If a new datastore is subsequently created on the same storage partition or LUN, the original metadata and data blocks are overwritten.

Symptoms and impact include:

• Virtual Machines previously residing on the datastore show as "Inaccessible" or "Orphaned" in vCenter.

• VMFS datastore appears empty or data is missing. 

• The underlying LUN was unmounted and then subsequently deleted from the vSphere inventory.

• Data recovery via software-based undelete tools is impossible once the blocks have been initialized for a new file system.

• ESXi hosts may report storage connectivity but 0 bytes of used space on the new volume.

 

Diagnostic Verification
To confirm if the original metadata has been overwritten, run the following command from the ESXi shell: 

vmkfstools -Ph -v 10 /vmfs/volumes/<datastore_name_or_uuid>

Expected Result: If the volume was recently recreated, the output will show a volume creation date/timestamp that does not match the original production datastore, confirming that the new file system has initialized and the original metadata is gone.

Environment

VMware vSphere 7.x

VMware vSphere 8.x

VMware vSphere 9.x

Cause

This issue occurs due to a two-step destruction of the filesystem:

1. Metadata Destruction: The Delete Datastore operation in vSphere explicitly destroys the VMFS volume metadata and the partition table on the storage device. Unlike a simple Unmount, this action clears the pointers to the data.
2. Physical Overwrite: Creating a new datastore initializes a new VMFS heartbeat and metadata structure on the same physical blocks. Once the new volume is formatted and metadata is written to the drive's beginning sectors, the original pointers to VMDK files are physically overwritten.
   Note that the datastore creation task does not provide a warning regarding the existing VMFS volume, as the partition was wiped during the Delete operation.

Resolution

Recovery via standard software-based partition tools is generally ineffective once the new volume has been formatted.
The only viable path for data recovery is restoration from backups. The following steps must be taken to restore service:

1. Stop all I/O: Immediately stop any operations on the newly created datastore to prevent further overwriting of underlying data blocks.

2. Storage-Level Recovery: Consult with your Storage Array vendor to determine if a "Snapshot" or "Clone" of the LUN exists that predates the deletion.

3. Restore from Backup:

   • Identify the latest valid backup of the impacted Virtual Machines.

   • Deploy a new datastore (if not already created).

   • Perform a full restore of the VMs from your backup solution (e.g., Veeam, NetBackup, Dell Avamar).

4. Inventory Cleanup: Right-click any "Inaccessible" VMs in the vCenter inventory and select Remove from Inventory.

   • Register the restored VMs from the new datastore.

 

Prevention and Best Practices

To avoid data loss during storage migrations or maintenance, follow these guidelines:

• Unmount Only: Use the Unmount operation exclusively if the intent is to temporarily remove storage without destroying data.
• Avoid Delete: Never select Delete Datastore unless the data is no longer required or has already been successfully migrated and verified on another volume.
• Storage Snapshots: Before performing storage maintenance, ensure a valid storage-array-level snapshot exists.

Additional Information

For further details on safe datastore management, see the following:

• Remove VMFS Datastores - TechDocs