Error: VMFS6 datastores remain in failed state after storage array recovery.

Products

• VMware vSphere ESXi

Issue/Introduction

Following a transient storage connectivity loss or array power failure, VMFS6 datastores may remain in an inaccessible or failed state even after physical paths are restored. Standard storage rescans typically fail to mount the volumes until the host is rebooted or stale kernel processes are cleared.

Symptoms:

• Datastores remain in a 'Failed' or 'Inaccessible' state after storage array power is restored.
• Virtual machines on affected datastores report disk loading errors or appear orphaned.
• A standard HBA adapter rescan (esxcli storage core adapter rescan --all) does not restore access.
• ESXi vmkernel.log shows the device flagged as APD Notify PERM LOSS or NMP Device is blocked despite active physical paths.
  
  vmkernel: cpu16:#######)StorageDevice: #####: Device naa.######## APD Notify PERM LOSS.

Environment

• VMware vSphere ESXi 8.x
• VMware vCenter Server 8.x

Cause

During an extended All Paths Down (APD) or Permanent Device Loss (PDL) event, active I/O threads and file system drivers may retain orphaned "storage worlds" (kernel process handles) in memory.

These stale locks prevent the Native Multipathing (NMP) plug-in from clearing the blocked state and prevent the Logical Volume Manager (LVM) from automatically re-probing and mounting the filesystem when connectivity returns.

Resolution

To recover the datastores without a host reboot, the user must manually terminate the orphaned kernel processes and force an LVM probe.

1. Identify the device ID (naa.####) for the affected datastore:
   esxcfg-scsidevs -m
   
   
2. List the running processes (worlds) associated with that device: 
   esxcli storage core device world list -d naa.#############


3. This command cannot be undone. Verify every parameter before running. Terminate the identified World IDs: 
kill -9 [WorldID]


4. Rescan the storage adapters: 
   esxcli storage core adapter rescan --all


5. Force a manual probe of the LVM metadata to mount the volumes: 
   vmkfstools -V

Note: If orphaned processes cannot be cleared, a host reboot is required to flush the kernel memory state.

Additional Information

• To speak with a customer representative or a Support Engineer, see Contact Support.
• Subscribe to this article to receive updates: How to subscribe to Knowledge Management articles.
• vCenter Server alarm "Cannot connect to Storage" and All Paths Down for a storage device
• Error: "Some of the disks of the virtual machine failed to load" and VM registration failure after storage connectivity loss.