vSphere HA agent unreachable due to storage lockup on ESXi host

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Administrators experience host unresponsiveness and vCenter Server alerts stating: "vSphere HA agent for this host has an error: The vSphere HA agent is not reachable from vCenter Server."
  
  
• The vSphere HA agent (FDM) fails to configure or connect despite verified, functional network connectivity.
  
  
• Attempting to navigate directories on the host via SSH results in Input/output errors:

[root@esx-host:~] cd /var/run/log
-sh: cd: can't cd to /var/run/log: Input/output error

• Checking dmesg reveals SCSI ATS miscompares and VMFS lock failures:

cpu37:2097280)ScsiDeviceIO: 4686: Cmd(0x45daadede780) 0x89, CmdSN 0x1eb454e from world 2103893 to dev "naa.624a..." failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0

cpu52:2103893)DLX: 5552: vol '<VOLUME_NAME>', lock at 133324800: Lock type: 10C00005. [Req mode 1] try lock error: Atomic test and set of disk block returned false for equality:

Environment

• VMware vCenter Server
• VMware vSphere ESXi 

Cause

• This issue is caused by a severe storage-level lockup and I/O hang on the ESXi host triggered by underlying Hardware Accelerated Locking (ATS) failures.
  
  
• When the storage array and the ESXi host experience an ATS heartbeat miscompare, the storage stack fails to acquire or release VMFS locks.
  
  
• This results in an I/O hang. Because I/O is completely frozen, the host experiences cascading failures: memory admission fails, internal processes ("worlds") are killed by the kernel, and the management agents (including the Fault Domain Manager / vSphere HA agent) completely freeze.

Resolution

1. Perform a reboot of the affected ESXi host. In case a graceful restart via vCenter or SSH is impossible, use the physical host's out-of-band management (iDRAC, iLO, CIMC) to perform a hard reboot.
   
   
2. Once the host boots back up, verify that it successfully reconnects to the datastores.
   
   
3. Check vCenter Server to ensure the vSphere HA agent initializes correctly, and the host rejoins the HA cluster. If necessary, right-click the host in vCenter and select Reconfigure for vSphere HA.