ESXi entering not_responding state and reconnecting to vCenter each minute after changing vCenter IP

Products

VMware vSphere ESXi

Issue/Introduction

In /var/log/vmware/vpxd/vxpd.log ESXi is regularly marked as not responding and reconnects almost instantly every minute:

YYYY-MM-DDTHH:35:37.938Z warning vpxd[30###27] [Originator@6876 sub=MoHost opID=HostSync-host-#####-28e##ca3] host [vim.HostSystem:host-#####,<hostname>] connection state changed to NO_RESPONSE
YYYY-MM-DDTHH:35:38.108Z warning vpxd[30###27] [Originator@6876 sub=MoHost opID=HostSync-host-#####-28e##ca3] host [vim.HostSystem:host-#####,<hostname>] connection state changed to CONNECTED
...
YYYY-MM-DDTHH:36:37.924Z warning vpxd[67###01] [Originator@6876 sub=MoHost opID=HostSync-host-#####-45d##ab7] host [vim.HostSystem:host-#####,<hostname>] connection state changed to NO_RESPONSE
YYYY-MM-DDTHH:36:38.112Z warning vpxd[67###01] [Originator@6876 sub=MoHost opID=HostSync-host-#####-45d##ab7] host [vim.HostSystem:host-#####,<hostname>] connection state changed to CONNECTED

Virtual Machines are marked as disconnected when ESXi is not_responding
Using the temporary workaround from KB ESXi host disconnects intermittently from vCenter Server to increase the heartbeat timeout results in the same issue, just at the new timeout interval (default 60 seconds)
vCenter IP was changed in the past
When doing KB Verify ESXi host heartbeat to vCenter using packet capture utilities to check heartbeating, vCenter receives no heartbeats and the destination IP from ESXi side is vCenter's old IP address

In /var/run/log/vpxa.log on ESXi at the time of each reconnect, the old vCenter IP is referenced:

Vpxa[799##03]: [Originator@6876 sub=vpxaInvtHost opID=HostSync-host-#####-73e3###7-1] Server IP: <old_IP_of_vCenter>  (id:83###1), host IP: <IP_of_ESXi>
Vpxa[799##03]: [Originator@6876 sub=Heartbeat opID=HostSync-host-#####-73e3###7-1] Started heartbeating..

vpxa config has the wrong server IP:

configstorecli config current get -c esx -g services -k vpxa_solution_user_config
{
"dvs": {
"dvs_sync_rate": 86400,
"port_sync_batch_limit": 100,
"portgroup_sync_batch_limit": 500
},
"host_ip": "<ESXi_IP>",
"host_key": "########-####-####-####-############",
"host_port": 443,
"server_ip": "<old_IP_of_vCenter>",
"server_port": 902,
"valid_user": "vpxuser"
}

Environment

vSphere 8.0 U3

Cause

The vCenter Database (VCDB) contains a stale management IP for the affected ESXi host in the vpx_host table. During reconnect, vCenter pushes this IP into the vpxa config.

ESXi will be marked as not_responding as it's sending UDP heartbeats to the wrong IP address and the vCenter it's actually connected with will not receive UDP heartbeats.

The ESXi will nearly instantly reconnect due to ongoing TCP connections from vCenter.

Resolution

Remove the old IP of vCenter from the VCDB:

Snapshot the vCenter, in accordance with KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice if vCenter is a member of ELM
Note the IDs of the issue ESXi hosts from the vCenter UI by:
1. Selecting the ESXi host in the UI
2. Taking the numeric portion of the host ID from the URL bar (format host-####)
SSH to vCenter with root account
Run the following to check the database for the old IP of vCenter replacing the hashes with the ID previously noted:
```
psql -U postgres -d VCDB -c "select id, dns_name, management_ip from vpx_host where id in (#####, #####);"
```
If the column "management_ip" contains the old IP vCenter address, continue the steps
Stop vpxd service:
```
service-control --stop vpxd
```

Update VCDB to set the management_ip to null:

psql -U postgres -d VCDB -c "update vpx_host set management_ip=null where id in (####, ####);"

Start vpxd service:
```
service-control --start vpxd
```
Disconnect the issue ESXi hosts in the vCenter UI
Reconnect the issue ESXi hosts in the vCenter UI

Additional Information

Note: When reconnecting the ESXi hosts in step 10, it may initially fail and then display the "add host" wizard; proceed with this wizard and provide root credentials when prompted