Multiple ESXi hosts are not responding or disconnecting and reconnecting to the managed vCenter Server

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Multiple ESXi hosts transition into a "not responding" state within the vCenter Server web interface.
Reconnection of the hosts fails with the following error message:

Cannot contact the specified host hostname\IP. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

This issue is indicative of DNS resolution failures. In the /var/log/vmware/vpxd/vpxd.log file of the vCenter Server, the following log snippets are observed:

YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=HostSync-host-####-a####a] Failed to resolve address; <resolver p:0x##########, '<ESXI-FQDN:443>', next:(null)>, e: 2(Host not found (non-authoritative), try again later), async: true, duration: 151515msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=PollQuickStatsLoop-#####] Failed to resolve address; <resolver p:0x##########, '<ESXi FQDN:443>', next:(null)>, e: 125(Operation canceled), async: true, duration: 168813msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=PollQuickStatsLoop-#####] Failed to resolve address; <resolver p:0x##########, '<ESXi FQDN:443>', next:(null)>, e: 125(Operation canceled), async: true, duration: 168813msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection opID=PollQuickStatsLoop-#####] Failed to resolve address; <resolver p:0x##########, '<ESXi FQDN:443>', next:(null)>, e: 125(Operation canceled), async: true, duration: 168816msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection] Failed to resolve address; <resolver p:0x##########, 'localhost:#####', next:(null)>, e: 125(Operation canceled), async: true, duration: 138509msec
YYYY-MM-DDTHH:MM:SS warning vpxd[#####] [Originator@6876 sub=IO.Connection] Failed to resolve address; <resolver p:0x##########, 'localhost:#####', next:(null)>, e: 125(Operation canceled), async: true, duration: 168509msec

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
VMware vSphere ESX 9.x

Cause

The vCenter Server attempts to reach all of the ESXi hosts in the inventory using their Fully Qualified Domain Names (FQDN). However, DNS resolution fails because the DNS server is down, unavailable, or unable to resolve one or more hostnames. This failure results in API calls that queue up or fail when processing host heartbeats.

Resolution

One of the following approaches must be utilized:

Option 1: Validate DNS Resolution from vCenter Server

Verification of the DNS server used by the vCenter Server is required to ensure it can successfully resolve the FQDN of all ESXi hosts, the vCenter Server, and vice-versa. If the DNS server IP address must be updated on the vCenter Server, see Update DNS Server IP address for vCenter Server.

Option 2: Manual Hosts File Modification

If the DNS server is down or permanently unavailable, a manual entry must be created in the /etc/hosts file of the vCenter Server for the affected ESXi hosts.

The /etc/hosts file must be updated with host entries utilizing the following format: <IP address> <hostname.example.com> <host_shortname>
Reconnection of the hosts from within the vCenter Server UI is required once the entries are saved.

For further details on safely editing this file, see Modifying host entry in /etc/hosts file for vCenter or ESXi host using vi editor.

Option 3: Verification of ESXi Host DNS Settings

The DNS configuration on the ESXi host level must be verified and updated. It must be ensured that the DNS server IP addresses and search domains configured on the host are accurate. These settings are required for the host to successfully resolve the vCenter Server FQDN and maintain consistent management traffic. For detailed instructions on updating these settings, see Configuring VMware ESXi Management Network from the direct console.