Unable to manage any VMs on the Host after failed attempt to unmount the Datastore

Products

• VMware vSphere ESXi

Issue/Introduction

A Datastore (FC) was unmounted from various Hosts by using the Web Client.

One or more of the following Symptoms are perceived:

• The unmount timed out on one of the Hosts, followed by the Host showing as "non-responding" in the Web Client.
• The Host crashed
• Restart of hostd does not resolve the issue
• Attempting to reconnect the Host in the Web Client fails with the error: "Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."

Environment

vSAN 8.x

Cause

The /var/log/vmkwarning.log is showing:
Wa(180) vmkwarning: cpu15:2097765)WARNING: Heap: 3645: Heap storageFPINHeap already at its maximum size. Cannot expand.
Wa(180) vmkwarning: cpu15:2097765)WARNING: Heap: 4105: Heap_Align(storageFPINHeap, 48/48 bytes, 64 align) failed.  caller: 0x420014d45425
Wa(180) vmkwarning: cpu15:2097765)WARNING: StorageFPIN: 521: Failed to allocate memory.

Resolution

Please follow KB 380321 to resolve the issue