Local datastore capacity reporting as 0B following physical disk reseat on ESXi Host

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host experienced a thermal event causing a local disk to report an "Amber" hardware status.
• After resolving the temperature and reseating the disk, the physical LED returned to "Green," but the associated local VMFS datastores reported 0B Capacity, 0B Provisioned, and 0B Free. Virtual machines, including vCenter Server, became inaccessible or hung due to the loss of backing storage.

Environment

• VMware ESXi 7.x, 8.x, 9.x

Cause

The issue is caused by a Metadata Desynchronization between the Physical Storage Layer and the ESXi Storage Stack. When the disk failed due to heat, the ESXi kernel marked the device paths as unavailable (APD/PDL). Although the physical path was restored by reseating the disk, the ESXi storage layer (PSA) did not automatically re-scan or re-mount the partition table, leaving the logical datastore object in a stale "0B" state in the host's memory.

Resolution

1. To manually rediscover storage devices and VMFS volumes via the ESXi command-line interface (CLI) without incurring downtime, execute the following commands in sequence:

   • Perform an HBA adapter rescan to detect new storage devices: esxcli storage core adapter rescan --all
   • Refresh the VMFS volumes to discover new datastores on those devices: vmkfstools -V

   Note: The vmkfstools -V command does not provide console output upon successful execution. To verify the results, you can check the VMkernel logs (/var/log/vmkernel.log) or list the current filesystems using esxcli storage filesystem list.

2. Powering off an ESXi host while virtual machines are actively running is a high-risk operation that can lead to guest operating system corruption, vCenter Server database (VCDB) inconsistencies, and permanent data loss. If a manual storage rescan or rediscover fails to resolve the issue, ensure all virtual machines are gracefully shut down or migrated before initiating a host power cycle.
3. If the vCenter Server health status is verified as Green and the management services are responsive, initiate a vMotion (Cross-Host Migration) for the resident Virtual Machines to an alternate host within the cluster to ensure service continuity

Steps to Migrate a VM using vMotion

• • Launch Migration Wizard: Right-click the virtual machine in the vSphere Client inventory and select Migrate.
  • Select Migration Type: Choose Change compute resource only (to move only the VM's execution to a different host).
  • Click Next.
  • Select Target Host: Select the destination ESXi host or cluster.
  • Review the Compatibility pane at the bottom to ensure it says "Compatibility checks succeeded."
  • Select Networks: Ensure the destination network matches the source. If they differ, select the appropriate port group on the target host.
  • Select vMotion Priority: Select Schedule vMotion with high priority (Recommended) for the fastest performance.
  • Finalize: Review the summary and click Finish.
  • Monitor the progress in the Recent Tasks pane.

       4. Perform a Cold Reboot after migrating all VMs: Power off the host completely and restart after migrating all the VMs. This forces the Storage Controller firmware to re-initialize and the ESXi kernel to perform a fresh discovery of all SCSI targets and VMFS volumes.

       5. Verify Volume Mounting: Post-reboot, log in to the ESXi Shell and verify the volumes are detected:

• • • esxcli storage vmfs extent list

 

6. Check Hardware Health: Verify the thermal status and disk health via the Integrated Dell Remote Access Controller (iDRAC), HP Integrated Lights Out (iLO), or equivalent BMC tool to ensure no permanent hardware degradation.

Additional Information

All Paths Down for a storage device

Permanent Device Loss (PDL) and All-Paths-Down (APD) on host