VMFS volume severely corrupted following power outage - VMware vSphere ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

This article addresses logical VMFS datastore corruption and inaccessibility in VMware vSphere environments following improper host shutdowns, power outages, or storage fabric instability.

Symptoms:

• Datastore displays 0MB capacity or "Cannot open volume" error after a power failure.
• vmkernel.log displays: WARNING: Vol3: 2423: [VolumeName] has severely corrupted.
• The vmkernel.log may also report: Failed to get object 28... No filesystem on the device.
• Fibre Channel HBA logs show Dropped frame(s) detected during the power instability window.

Environment

• VMware vSphere ESXi 7.x / 8.x
• External SAN storage with QLogic HBAs.

Cause

Sudden power loss or storage interruptions cause "torn writes", where metadata updates to the VMFS filesystem are only partially committed. This results in logical metadata corruption that prevents the ESXi host from safely mounting the volume for I/O operations.

Resolution

Perform the following steps to restore datastore access:

1. Identify the most recent healthy storage-level snapshot taken prior to the power outage.
2. Evacuate all accessible surviving workloads and unmount the corrupted datastore from all ESXi hosts in the cluster.
3. To recover remaining VMs that could not be migrated, revert the backend LUN to the identified pre-outage storage snapshot on your storage array.
4. Perform a storage rescan on the ESXi cluster to detect and remount the restored volume.
5. If you are unable to perform steps 1-4 due to unavailable or corrupted storage snapshots, contact Broadcom Support to attempt to repair the VMFS file system by running VOMA.
   Note: To run VOMA, datastore must be in an unmounted state.

 

Additional Information

• For steps on how to collect VMware ESXi logs, see How to collect diagnostic information.