ESXi hosts show intermittent Not Responding status in a vSphere cluster

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts in the cluster intermittently show a "Not Responding" status in the vSphere Client.
  
  
• Virtual machines (VMs) stored on SAN LUNs remain operational, but performance on the affected ESXi hosts decreases significantly.
  
  
• Host service restarts fail to resolve the state.
  
  
• Examining the /vmfs/volumes directory reveals four BOOTBANKs (two instances of BOOTBANK1 and two instances of BOOTBANK2).
  
  
• Additionally, two OSDATA volumes mount to the system. This condition typically indicates a duplicate boot LUN presentation.

Environment

• VMware vSphere ESXi 7.x

• VMware vSphere ESXi 8.x

Cause

A storage device incorrectly functions as a Boot LUN for multiple ESXi hosts, making the device accessible across the cluster. Presenting multiple Boot LUNs to a single ESXi host creates ambiguity regarding the legitimate OSDATA. This storage conflict causes the hostd process to fail, leading to the "Not Responding" symptom.

Resolution

1. Identify the correct BOOTBANK and OSDATA volumes before removal:
   
   • Run esxcli storage core device list | grep -E "Display Name|Is Boot Device" to confirm the device currently booting the host.
     
     
   • For each rogue BOOTBANK or OSDATA UUID listed in the esxcli storage filesystem list output, run vmkfstools -P /vmfs/volumes/<UUID> to identify the specific naa. or mpx. identifier and partition number.
     
     
   • Verify active system paths and confirm the UUIDs of the active system partitions by checking the symlinks via ls -l / for bootbank, altbootbank, and scratch.
2. Unpresent the duplicate LUNs from the cluster at the storage array level.
   
   
3. Verify if the duplicate OSDATA volumes remain visible under the /vmfs/volumes directory.
   
   
4. Reboot the affected ESXi hosts if the volumes remain visible to restore normal management operations.

Additional Information

Ensure that boot LUNs are masked and zoned exclusively to the specific host they are intended to boot. Broadly presenting boot LUNs to an entire cluster leads to volume mounting conflicts and management service failures.