ESXi hosts in a HA cluster randomly display the message: A possible host failure has been detected on in cluster in datacenter

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• In the Summary tab of the ESXi host in a HA cluster, you see this message:
  A possible host failure has been detected on <host> in cluster <cluster> in <datacenter> datacenter

 

• Reconfiguring HA on the affected host temporarily corrects this issue.
• In the /var/log/vmware/aam/vmware_<hostname>.log file of one or moreESXi hosts, you see entry similar to:
  MESSAGE: Message received from an unexpected sender: msgid 621 from (1/0:rtcp_1.0)

 

• In the /var/log/vmkernel.log log file of the ESXi host, you see entry similar to:
  Tcpip_Vmk: 143: arp: xx:xx:xx:xx:xx:xx is using my IP address <esxi_ip>!
warning 'HostsvcPlugin'] Duplicate IP <esxi_ip> detected at mac xx:xx:xx:xx:xx:xx

 

• In the /var/run/log/vobd.log file of the ESXi host, you see entry similar to:
  <timestamp>  [netCorrelator] 5716995306931us: [vob.net.vmknic.ip.duplicate] A duplicate IP address was detected for <esxi_ip> on interface vmk0. The current owner is xx:xx:xx:xx:xx:xx
<timestamp>  [netCorrelator] 5717119985527us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for <esxi_ip> on interface vmk0, current owner being xx:xx:xx:xx:xx:xx

 

• in vCenter Server vpxd.log file ypou may see lines similar to:
  <timestamp>  warning vpxd[08752] [Originator@6876 sub=IO.Connection opID=<op_id>] Failed to SSL handshake; SSL(<io_obj p:0x00007fd88cd02178, h:248, <TCP '<VCENTER_IP> : 56374'>, <TCP '<ESXI_IP> : 443'>>), e: 336134278(certificate verify failed), duration: 1msec
 Exception: Verification parameters:
--> PeerThumbprint: <UNEXPECTED_THUMBPRINT>
--> ExpectedThumbprint: <ESXI_THUMBPRINT>
--> ExpectedPeerName: <ESXI_HOSTNAME>
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)

Environment

VMware vSphere ESXi 6.*
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
VMware vSphere vCenter 6.*
VMware vSphere vCenter 7.0
VMware vSphere vCenter 8.0

Cause

This issue occurs when a device on the network is configured with the same IP address as one of the ESXi hosts in a HA cluster. In such a case:

• The HA agent receives invalid data and generates errors
• The HA agent does not function properly until it is reconfigured
• The frequency of this issue depends on the IP conflict encountered

Resolution

To resolve this issue, find the device with the duplicate IP address and configure it with a different IP address.

You can find the Duplicate IP and MAC address in the /var/log/vobd.log log file of the ESXi host to identify the device with the duplicate IP address.

Note: Duplicate IP may or may not be available in the messages or vobd.log. In such a case, reconfigure HA at the cluster or host level.