ESXI Hosts Disconnected in vSphere Following DNS Server Snapshot Reversion
Article ID: 435196
Updated On:
Products
Issue/Introduction
All ESXI hosts display a disconnected or not responding state within the vCenter Server inventory.
Manual attempts to reconnect the hosts fail and time out.
ESXI hosts are added to the vSphere inventory by FQDN.
Network connectivity (ping) between vCenter Server and the ESXI hosts via IP address is successful, but FQDN resolution fails.
This issue typically occurs following a network disruption or snapshot revert of the primary Domain Controller or DNS server.
DNS is running on the Domain Controller that was reverted to a snapshot.
Environment
- VMware vCenter Server
- VMware vSphere ESXI
Cause
ESXI hosts disconnect from the vCenter Server when FQDN resolution fails. In this scenario, the vCenter Server is unable to resolve hostnames due to an unreachable DNS server. The outage is triggered when the primary Domain Controller (acting as the DNS server) is reverted to a snapshot, resulting in the virtual machine's network adapter being connected to an incorrectly configured or isolated ephemeral port group.
Due to the host being in a disconnected/not responding state, the virtual machine's network adapter cannot be moved back to the correct distributed port group until the host on which the Domain Controller resides is brought back online in vSphere.
Resolution
1. Validate the network connectivity and status of the environment's DNS server or Domain Controller. If a snapshot revert was recently performed, confirm the virtual machine's network adapter is assigned to the correct distributed port group.
2. If the DNS server cannot be immediately restored, temporarily bypass DNS by adding local host file entries directly on the affected ESXi hosts using the following command via SSH:
esxcli network ip hosts add -H <HOSTNAME> -I <IP_ADDRESS>
3. Add the corresponding host entries to the vCenter Server by editing the /etc/hosts file via SSH.
4. Clear any hung reconnection tasks in the vCenter Server by manually setting the hosts to a disconnected state in the vCenter database. SSH into the vCenter Server appliance and execute:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled = 0;"
5. Restart the vCenter Server vpxd service to apply the database changes.
6. Log in to the vSphere Client UI and manually connect the disconnected ESXI hosts.
7. Restore the primary Domain Controller/DNS server to the correct network port group.
8. Once DNS services are fully restored, remove the temporary host file entries from both the ESXI hosts and the vCenter Server to resume standard DNS operations.
Additional Information
To prevent widespread host disconnections during maintenance or component failures, it is recommended that redundant Domain Controllers and DNS servers are deployed and accessible to the management domain. Reverting a Domain Controller virtual machine to a snapshot is not recommended.
Feedback
