Backup Connection Refused (TCP/902) Due to ESXi vmkernel IP Conflict

Products

• VMware vSphere ESXi

Issue/Introduction

• Backups fail with "Connection refused" on TCP/902 to the ESXi host, despite successful pings to the ESXi host's IP
• Backups fail for VMs on a specific ESXi host.
• netcat -vz <ESXi_IP> 902 or telnet shows "Connection refused."
• ping <ESXi_IP> from the backup appliance succeeds, but TCP/902 fails.
• Intermittent resolution observed when manually pinging from the ESXi host.

Environment

 

• vSphere ESXi 7.x
• vSphere ESXi 8.x
• vSphere vCenter Server 6.x
• vSphere vCenter Server 7.x
• vSphere vCenter Server 8.x

Cause

• A duplicate IP address conflict exists between the ESXi host's vCenter Agent (VMkernel vmk0/vmkX) and a network interface on one of the nodes within the backup appliance itself. This conflict causes the physical network's ARP tables to intermittently map the single IP to two different MAC addresses, leading to traffic misdirection. 
• The Conflict: The same IP address is configured on the ESXi host's VMkernel interface (e.g., vmk0) and on a network interface belonging to a backup appliance node.
•  ARP Table Poisoning: When the backup appliance (or any device) sends an ARP request for this IP, both the ESXi host and the backup appliance node will respond. The physical switches (and the backup appliance's ARP table) will then intermittently store either MAC address for that single IP. 
• Misdirected Traffic: * Ping Success: Pings may succeed because the ESXi host's VMkernel or the duplicate backup appliance node might respond to ICMP. When an ESXi host initiates a ping, the switch learns its MAC correctly, temporarily resolving the ARP entry. 
  * TCP/902 Failure: The TCP connection for backups (port 902) requires communication specifically with the ESXi host's vmk interface, which has hostd (or vpxa) listening on port 902. When the backup appliance attempts to connect to TCP/902, if the physical network routes the traffic to the duplicate IP on the backup appliance node (which is not an ESXi host and doesn't have hostd listening on port 902), the connection is immediately rejected, resulting in "Connection refused."
• Packet Capture Evidence: A packet capture on the ESXi host's UplinkRcvKernel showed no packets received from the backup appliance during the TCP/902 test, confirming traffic was never reaching the ESXi host. 
  

Resolution

Assign a unique and unused IP address to the ESXi host's VMkernel interface configured for backups. 

1. Identify the Conflicting IP: Locate the exact IP address that is duplicated. 
2. Assign New IP: Reconfigure the ESXi host's VMkernel interface (vmk) for backups with a new, unique IP address within the same subnet. 
3. Refresh Network Nodes: Ensure the backup appliance and other network devices refresh their ARP tables. 
4. Verify Connectivity: * ping <new_ESXi_IP> from the backup appliance. * netcat -vz <new_ESXi_IP> 902 from the backup appliance. * Confirm only a single MAC address is associated with the new IP in the backup appliance's ARP table.