ESXi host frequently getting disconnected or unresponsive on vSphere client due to IP conflict

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• An ESXi host frequently becomes unresponsive and disconnects from the vCenter Server UI, even when ports 80, 443, and 902 are open between the vCenter Server and the affected host. This can be verified by runing below commands:
  
  On vCenter Server :
curl -v telnet://<Affected_ESXI_Mgmt_IP>:<port#>
  
  On ESXi host: 
  nc -z <vcenter_ip> <port#>
nc -zu <vcenter_ip> <902>
  
  
• Using nslookup, forward and reverse name resolutions work on both the vCenter Server and the affected host and they are able to resolve each other's FQDNs and IP addresses:
  
  nslookup <IP/FQDN>
  
  
• The following error is encountered in the vCenter Server UI when attempting to reconnect the affected host:
  
  Disconnected from host. Reason: Cannot verify the SSL trust


• In the /var/run/log/vobd.log file of the affected ESXi hosts and other hosts in the cluster, entries similar to the following indicate an IP conflict with the affected ESXi host's management IP address:

YYYY-MM-DDThh:mm:ss In(##) vobd[#######]:  [netCorrelator] ########us: [vob.net.vmknic.ip.duplicate] A duplicate IP address was detected for <Affected_ESXI_Mgmt_IP> on interface vmk0. The current owner is <Duplicate_IP_owner_MAC>.
YYYY-MM-DDThh:mm:ss In(##) vobd[#######]:  [netCorrelator] ########us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for <Affected_ESXI_Mgmt_IP> on interface vmk0, current owner being <Duplicate_IP_owner_MAC>.
YYYY-MM-DDThh:mm:ss In(##) vobd[#######]:  The event ([esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for <Affected_ESXI_Mgmt_IP> on interface vmk0, current owner being <Duplicate_IP_owner_MAC>) was sent immediately to hostd;

Cause

Another node (virtual machine, ESXi host, or other network device) on the cluster's management network uses the same IP address as the affected ESXi host, resulting in a duplicate IP address conflict.

Resolution

To resolve the IP conflict, follow the below steps

1. Identify the node or device that currently owns the duplicate IP address on the management network. Work with your network team and use the MAC address obtained from the vobd.log file to locate the conflicting node.

2. Remove or reconfigure the conflicting node with a different, available IP address.

3. If reconfiguring the conflicting node is not feasible, perform the following steps to change the affected ESXi host IP:

   1. Remove the affected ESXi host from the vCenter Server inventory.

   2. Connect directly to the affected ESXi host via the Direct Console User Interface (DCUI).

   3. Change the management IP address to a new, unused static IP address, or configure it to use DHCP.

   4. Add the ESXi host back to the vCenter Server inventory.

Additional Information

Changing the ESXi server management IP address

Duplicate IP address detected