ESXi constantly receiving "Agent can't send heartbeats: Host is down" events

Products

• VMware vSphere ESXi

Issue/Introduction

Hosts are constantly receiving "Agent can't send heartbeats: Host is down" events.

 

In ESXi logs you see following 

 

/var/run/log/clusterAgent.log

clusterAgent[2100008]: WARN grpc: addrConn.createTransport failed to connect to {#####.#####.####:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
grep: crx: Is a directory
clusterAgent[2100008]: WARN grpc: addrConn.createTransport failed to connect to {#####.#####.####:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...

 

/var/run/log/watchdog.log

watchdog[2104941]: Started etcdmain with PID=4361661
watchdog[2104941]: Restarting etcdmain
watchdog[2104941]: Started etcdmain with PID=4381390
watchdog[2104941]: Restarting etcdmain

 

Investigating log entries you ruled out following articles as possible cause and solutions

"Admission failure in path: host/vim/vmvisor/etcd:etcd", frequent Etcd crash on the ESXi host causing too many DNS queries

ESXi Hosts Constantly Sending DNS Queries to the DNS Server

Environment

VMware vSphere ESX 8.x

Cause

Due to issues in environment network(faulty switch) hosts couldn't properly communicate between each other hence triggering the events.

Resolution

Engage your internal network team for review of network settings and hardware.