ESXi host isolation and unresponsiveness during concurrent management tasks

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Maintenance mode tasks (e.g., Enter Maintenance Mode) hang indefinitely.
• The vCenter Server /var/log/vmware/vpxd/vpxd.log file reports "Null response" or "Service Unavailable" exceptions for the affected ESXi host.
  YYYY-MM-DDThh:mm:ss.###Z error vpxd[####] [Originator@#### sub=MoHost opID=######] Null response from host [vim.HostSystem:host-##,<host_fqdn>]; no error
YYYY-MM-DDThh:mm:ss.###Z error vpxd[####] [Originator@#### sub=Vmomi opID=######-#####-auto-das-h5:######-##] Got vmacore exception when invoking VMOMI method; <</hgw/host-##>, /vpxa>, vpxapi.VpxaService.setServer, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
  
  
• SSL handshake failures appear in the ESXi /var/run/log/hostd.log file.
  YYYY-MM-DDThh:mm:ss.###Z Wa(##) Hostd[####]: [Originator@#### sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x000000#####, h:##, <TCP '127.0.0.1 : #####'>, <TCP '127.0.0.1 : 80'>>), e: #######(certificate verify failed (SSL routines)), duration: #msec
YYYY-MM-DDThh:mm:ss.###Z Wa(##) Hostd[####]: [Originator@#### sub=HttpConnectionPool-####] Failed to get pooled connection; <cs p:000000####, TCP:localhost.localdomain:##>, SSL(<io_obj p:0x000000####, h:##, <TCP '127.0.0.1 : #####'>, <TCP '127.0.0.1 : 80'>>), duration: #msec,


• The host is marked as disconnected in the vCenter Server.
• The High Availability (HA) primary host initiates FDM recovery procedures (datastore heartbeats/ICMP pings) due to host unreachability in the /var/run/log/fdm.log file.
  YYYY-MM-DDThh:mm:ss.###Z Er(##) Fdm[####]: [Originator@#### sub=Cluster opID=clusterManager.cpp:###-####] Dead connection for slave @ host-##
YYYY-MM-DDThh:mm:ss.###Z Db(##) Fdm[####]: [Originator@#### sub=Cluster opID=clusterManager.cpp:###-####] Forcing heartbeat check on datastore /vmfs/volumes/<volume-id> for slave host-##
YYYY-MM-DDThh:mm:ss.###Z Db(##) Fdm[####]: [Originator@####sub=Cluster opID=clusterManager.cpp:###-####] Beginning ICMP pings every 1000000 microseconds to host-##

Environment

• vCenter Server 8.0.x
• vSphere ESXi 8.0.x

Cause

• Concurrent management tasks (such as simultaneous vMotion, maintenance mode requests, or manual disconnect commands) can overwhelm the ESXi host management services, specifically the hostd and vpxa services.
• This resource contention may lead to management service unresponsiveness, subsequent SSL verification failures, and ultimately, an HA-level isolation event where the host is marked as "FDMUnreachable."

Resolution

To prevent host isolation and task failure, perform management tasks sequentially rather than concurrently:

1. Ensure all active vMotion, maintenance mode, or host configuration tasks are completed before initiating new management operations.
2. Monitor the task status in the vCenter Server UI to confirm completion before beginning the next operation.
3. Avoid automated processes that may trigger conflicting management events on the same ESXi host.

Additional Information

• If a host becomes isolated, check for HA recovery events in the primary HA host /var/run/log/fdm.log file.
• Review the hostd and vpxa logs for SSL handshake errors or service crashes corresponding to the time of the maintenance operation.
• Reference the VMware Ports and Protocols page to ensure the network configuration is not contributing to communication loss.