ESXi Host disconnected from vCenter following Upgrade failure

Products

• VMware vSphere ESXi

Issue/Introduction

• Following an ESXi host upgrade failure or subsequent rollback attempts post the upgrade failure, the host enters a "Disconnected" state in vCenter Server.
  
  
• Attempts to reconnect the host fail with the error below :-
  
  "Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."
  
  
• Bidirectional communication (ICMP and TCP 443/902) is broken between the vCenter Server and the specific ESXi Management IP. Refer Troubleshooting network and TCP/UDP port connectivity issues on Hosts
  
  
• The host and vCenter can both ping their respective default gateways, but cannot reach each other.
  
  
• When ICMP ping is performed from vCenter to the ESXi host, the pktcap-uw packet captures show the ICMP reply is sent by the desitnation ESXi host management VMkernel adapter, and the specific ICMP reply is going out of the VMNIC physical NIC adapter where the host management is pointed to, however, the packets are not received on the VMNIC of the ESXi host where vCenter is. 

From source vCenter's vmnic (physical nic adapter), the ICMP request packets are going out 

On the destination ESXi host management vmnic (physical nic adapter), the ICMP requests are being received and corresponding ICMP responses are going out.

The ICMP responses are not seen on the vCenter vNIC 

• The issue persists even after a clean re-installation of the ESXi operating system using the original IP address.

Environment

VMware vSphere ESXi

Cause

• As the pktcap-uw captures prove the ESXi host is successfully pushing the ICMP reply out of its physical NIC (vmnic), the host itself is functioning perfectly.
• The drop is happening at the physical switch layer or underly network as the ICMP reply is not seen on the physical NIC of the ESXI host where the vCenter is running on.

 To know more about running pktcap-uw command, refer Packet capture on ESXi using the pktcap-uw tool

Resolution

It is highly recommended to engage the physical network team to perform a deep-dive investigation to validate the ARP/MAC tables, MAC Bindings/ARP caches and checking the firewall session states etc, in knowing why the original IP of the ESXi host is blackholed.

Additional Information

Please note :- Before implementing the below workaround, ensure that ESXi host is not a part of the DVS, NSX/VSAN cluster, or configured in any of 3rd party monitoring tools.

As a workaround and to immediately restore the management connectivity of the Stand-Alone ESXi host, the ESXi management interface can be migrated to a new, functional IP address that is not used in the network.

1. Identify a Free IP: Select a new, unused IP address within the same management subnet/VLAN.
2. Test Connectivity: Temporarily assign the new IP to a test interface or the host to verify that bidirectional communication (ping/HTTPS) with the vCenter Server is successful.
3. Reconfigure ESXi Management:
   • Access the host via the Direct Console User Interface (DCUI) or ESXi Shell.
   • Update the Management Network configuration with the new IP address, subnet mask, and gateway.
4. Update DNS Records:
   • Update the Forward (A) record to map the existing host FQDN to the new IP address.
   • Update the Reverse (PTR) record to ensure proper resolution.
5. Reconnect to vCenter: In the vSphere Client, right-click the disconnected host and select Connection > Connect. Ensure the host reconnects successfully using the updated IP (via FQDN).