All ESXi not responding or disconnecting & reconnecting to their managed vCenter Server

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Multiple ESXi hosts are going into a not responding state in the vCenter web interface.
• Reconnecting the hosts fails with error:
  
  Cannot contact the specified host hostname\IP. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.
• In the /var/log/vmware/vpxd/vpxd.log of the vCenter Server, below log snippets are found:
  
  YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=HostSync-host-####-a####a] Failed to resolve address; <resolver p:0x##########, '<ESXI-FQDN:443>', next:(null)>, e: 2(Host not found (non-authoritative), try again later), async: true, duration: 151515msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=PollQuickStatsLoop-#####] Failed to resolve address; <resolver p:0x##########, '<ESXi FQDN:443>', next:(null)>, e: 125(Operation canceled), async: true, duration: 168813msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=PollQuickStatsLoop-#####] Failed to resolve address; <resolver p:0x##########, '<ESXi FQDN:443>', next:(null)>, e: 125(Operation canceled), async: true, duration: 168813msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=PollQuickStatsLoop-#####] Failed to resolve address; <resolver p:0x##########, '<ESXi FQDN:443>', next:(null)>, e: 125(Operation canceled), async: true, duration: 168816msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection] Failed to resolve address; <resolver p:0x##########, 'localhost:#####', next:(null)>, e: 125(Operation canceled), async: true, duration: 138509msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection] Failed to resolve address; <resolver p:0x##########, 'localhost:#####', next:(null)>, e: 125(Operation canceled), async: true, duration: 168509msec

Environment

• vCenter 7.x
• vCenter 8.x
• vCenter 9.x
• ESXi 7.x
• ESXi 8.x
• ESXi 9.x

Cause

• vCenter Server is attempting to reach all of the hosts in vCenter using their FQDN but the DNS server is unable to resolve one or more host name leading to API calls that queue up or fail in processing host heartbeats.
• DNS Server is down or unavailable.

Resolution

Option 1: Ensure the DNS Server used by vCenter are able to resolve the FQDN of all the ESXi hosts, vCenter and vise-versa.

Option 2: If the DNS Server is down or unavailable, then create a manual entry in /etc/hosts file of the vCenter server for the ESXi hosts in the below format and reconnect the Hosts in the vCenter server UI. For more information, refer Modifying host entry in /etc/hosts file for vCenter or ESXi host using vi editor

                <IP address> <hostname.example.com> <host_shortname>