Host connection and power state alarm in vCenter

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Host connection and power state alarm reported in vCenter.
  
  
  
  
• ESXi host's reports "Not Responding" in vCenter.
  
  
• The vpxd.logs at /var/log/vmware/vpxd/ shows missing heartbeats timed out between vCenter & ESXi hosts.
  
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[06805] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-95b3986] [VpxdHostCnx] Host connection reaper thread exiting
  YYYY-MM-DDTHH:MM:SS.SSSZ warning vpxd[06805] [Originator@6876 sub=VpxProfiler opID=CheckforMissingHeartbeats-95b3986] CheckforMissingHeartbeats [TotalTime] took 1802620 ms
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[69921] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-2b1cd032] [VpxdHostCnx] Host connection reaper thread exiting
  YYYY-MM-DDTHH:MM:SS.SSSZ warning vpxd[69921] [Originator@6876 sub=VpxProfiler opID=CheckforMissingHeartbeats-2b1cd032] CheckforMissingHeartbeats [TotalTime] took 541110 ms
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[26340] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-7fafaeb8] [VpxdHostCnx] No heartbeats received from host; cnx: 525c####-1###-3###-9###-#######, h: host-14, time since last heartbeat: 66183ms
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[26340] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-7fafaeb8] Marking the connection alive to false: 525c####-1###-3###-9###-#######
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[26340] [Originator@6876 sub=InvtHostCnx opID=CheckforMissingHeartbeats-7fafaeb8] Got lost connection callback for host-id
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[26340] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-7fafaeb8] [VpxdHostCnx] No heartbeats received from host; cnx: 525c####-1###-3###-9###-#######, h: host-id, time since last heartbeat: 126190ms
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[26340] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-7fafaeb8] [VpxdHostCnx] No heartbeats received from host; cnx: 525c####-1###-3###-9###-#######, h: host-id, time since last heartbeat: 186196ms
  
  

Environment

VMware vCenter Server

VMware ESXi

Cause

'Network latency' between ESXi and vCenter can affect their connectivity, ultimately causing the hosts to appear in a "Not Responding" state.

Resolution

ESXi host and vCenter connectivity can be affected by network issues, particularly latency, which can lead to missed heartbeats. To diagnose network and latency problems, packet captures and traceroutes should be performed on the physical infrastructure.

To perform the packet capture on ESXi, please refer - Packet capture on ESXi using the pktcap-uw tool

Additional Information

Host connection and power state" alarm seen on many or all hosts in vCenter

Japanese KB: vCenter 上で "ホストの接続と電源状態" アラームが発生する