ESXi hosts disconnecting after changing IP address of vCenter

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Managed ESXi hosts get disconnected or enter Not Responding state for 90 seconds repeatedly.
  
  
• vCenter Server does not receive heartbeats from the ESXi hosts even though 902 UDP port is open between ESXi hosts and vCenter IP address
  
  
• The VirtualCenter.AutoManagedIPV4 parameter from vCenter object > Configure > Advanced Settings may point to a different IP address than what is assigned to the vCenter 
  
  
• Attempting to update the vCenter IP address on ESXi config using below configstorecli commands and then reviewing it shows that it keeps reverting to the previous vCenter IP address
  
  configstorecli config current set -c esx -g services -k vpxa_solution_user_config --path /server_ip --value "<vcenter_ip>"
configstorecli config current get -c esx -g services -k vpxa_solution_user_config --path /server_ip
  
  
• Connecting to the vCenter Server database using command /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres and executing the below query shows the management_ip and local_ip_address columns dont contain the current vCenter IP address
  
  select * from vpx_parameter where name like '%Managed%';
select id, dns_name, management_ip, local_ip_address from vpx_host;

Environment

vCenter 7.x, 8.x, 9.x

Cause

The issue occurs in scenarios where the newly updated ip address of the vCenter which was edited via Management Interface ( VAMI ) did not get properly updated in the vCenter database. 

Resolution

To resolve the issue, update the vCenter current IP address on the VCDB by following below steps:

1. Take a snapshot/backup of the vCenter VM (In case the vCenter is in linked mode, take powered off snapshots on all ELM nodes)
   
   
2. Stop the vpxd service
   
   service-control --stop vmware-vpxd
   
   
3. Connect to the VCDB
   
   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
   
   
4. Run the below queries to update the old IP with the new, desired IP (where #.#.#.# is the new IP address of vCenter)
   
   update vpx_parameter set value = '#.#.#.#' where name = 'VirtualCenter.AutoManagedIPV4';
update vpx_parameter set value = '#.#.#.#' where name = 'VirtualCenter.ManagedIP';
update vpx_host set management_ip = '#.#.#.#' where id > '0';
update vpx_host set local_ip_address = '#.#.#.#' where id > '0';
   
   
5. Exit the VCDB database
    
   \q
   
   
6. Start the vpxd service
   
   service-control --start vmware-vpxd
   
   
7. In case the ESXi hosts still do not get connected back automatically, disconnect and reconnect the affected hosts manually.

Additional Information

Verifying the vCenter Server Managed IP Address

Verify ESXi host heartbeat to vCenter using packet capture utilities