ESXi host showing not responding in vCenter cluster due to physical switch(s) reboot

Products

• VMware vSphere ESXi 8.0
• VMware vCenter Server

Issue/Introduction

• The ESXi host in the vSphere UI, shows as not responding.
• Physical switch(s) directly connected to the ESXi host(s) and/or the ESXi host with the vCenter appliance or in the path, have been rebooted.
• In the logs of the impacted ESXi host, /var/run/log/vpxa.log, the following errors are seen:

error vpxa[1234567] [Originator@1234 sub=Heartbeat opID=SWI-####] Agent can't send heartbeats: Host is down

• On impacted host(s) prepared for NSX, BFD (Bidirectional Forwarding Detection) is down or flapping, as can be seen in /var/run/log/vmkernel.log:

cpu10:2098392)BFD_HandleStatusChange:859:[nsx@6876 comp="nsx-esx" subcomp="bfd"]local: 10.1.1.7, remote: 10.1.1.5, oldState: up, newState: down, diag: Neighbor Signaled Session Down, type: overlay

cpu32:2098393)BFD_HandleStatusChange:859:[nsx@6876 comp="nsx-esx" subcomp="bfd"]local: 10.1.1.7, remote: 10.1.1.5, oldState: up, newState: down, diag: Control Detection Time Expired, type: overlay

• Host prepared for NSX shows connections to controllers down, as can be seen in the /var/run/log/vmkernel.log:

cpu0:2101457)vdl2: VDL2CPProcessLinkChange:6891: [nsx@6876 comp="nsx-esx" subcomp="vdl2-1234567"]Control plane link down[IP: 10.1.2.1] for VNI[#####]

Note: IP addresses 10.1.1.7 and 10.1.1.5 above are TEP (Tunnel End Point) addresses assigned to the NSX transport node (ESXi hosts) and IP address 10.1.2.1, is the IP address of an NSX manager.

Environment

vSphere

Cause

The host is showing as unresponsive in the vSphere UI, because it is unable to send the heartbeats to vCenter.

Resolution

Plan to reboot networking devices such as switches and router in a maintenance window to avoid unforeseen impact to the ESXi hosts and workload on the ESXi hosts.