ESxi hosts are intermittently not responding in Cluster

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts in the cluster intermittently show as "Not Responding" in the vSphere Client. Despite this, the virtual machines (VMs), which are stored on a SAN LUN, remain operational. However, performance is significantly impacted on the affected ESXi hosts.

Restarting services on the ESXi hosts does not resolve the issue.

Environment

vSphere 7.x 

vSphere 8.x

Cause

Upon examining the /vmfs/volumes directory, it was observed that there were four BOOTBANKs (2 instances of BOOTBANK1 and 2 instances of BOOTBANK2). Additionally, two OSDATA volumes were mounted.

Resolution

The device is configured as a boot LUN for certain hosts and should not be accessible by other hosts within the cluster.

1. Instruct the array team to unpresent the LUNs from the cluster, as this is affecting multiple ESXi hosts.

2. After the boot LUN has been unpresented (and should no longer be zoned to the hosts), verify whether the two OSDATA volumes are still visible under /vmfs/volumes.

3. If the volumes remain visible, a reboot of the affected ESXi hosts may be required to restore normal operation.

Additional Information

Presenting two Boot LUNs to a single ESXi host could result in corruption, as the host would be unable to determine which LUN is the 'legitimate' OSDATA (scratch) LUN to write to. This ambiguity could cause issues such as the hostd process not responding, as observed in the symptoms.