Host connection and power state alarms for all hosts in the vCenter

Products

• VMware vCenter Server

Issue/Introduction

• Host connection and power state alarms for all hosts in the vCenter.
• vCenter services like vmware-updatemgr are not starting after a vCenter services restart.
• vCenter cannot forward/reverse lookup the FQDN/IP addresses of the ESXI hosts, this is confirmed by running nslookup IP or nslookup FQDN from the vCenter SSH.
• However vCenter is able to curl to the ESXI hosts successfully over ports 443 and 902, for example curl -v telnet://ESXiIPAdress:443
• Checking /var/log/vmware/vmware-sps/sps.log reports "flush span" messages along with messages like failed to log into vpxd.
• The following steps can be used as a proof of concept where the vCenter should function as normal when the hosts are disconnected:
   - Disconnect each host from the vCenter database, refer to KB: Disconnect hosts from vCenter via database when vCenter fails to start
   - Restart the vCenter services once all hosts are disconnected: service-control --stop --all && service-control --start --all

Environment

vCenter 8.x

ESXi 8.x

Cause

DNS resolution from vCenter targeting the ESXI hosts is not working.

Resolution

1. Take a snapshot of the vCenter before proceeding.
2. Update the vCenter DNS settings to use another DNS server that is reachable by following KB: Update DNS Server IP address for vCenter Server Appliance
   - Note that the VAMI may not work hence the DCUI of the vCenter VM will need to be used.
3. Reconnect each ESXI host to the vCenter via the vCenter UI.