ESXi backup fails with VDDK Error 14009: Port 902 blocked between Data Protection Cluster and ESXi host

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

When performing VM-level backups from a Data Protection Cluster, the backup process fails even if ports(902) are open. This issue often occurs intermittently or for specific ESXi hosts while other hosts in the same environment function correctly.

• Backup jobs fail consistently for specific hosts.
• No active TCP connections on port 902 are visible on the ESXi host during the backup attempt.

Error Message on Backup Appliance

[kVixError]: [1-4-212] [Code 14009] The server refused connection. Falling back to CBT.
kVixError error_msg: "[1-4-212] [Code 14009] The server refused connection

Environment

VMware ESXi 7.x, 8.x, 9.x

VMware vCenter Server

Cause

A network firewall or security rule is blocking TCP port 902 traffic between the backup cluster nodes and the ESXi host. In multi-node backup clusters, the firewall might only permit traffic for a single primary node, causing failures when other nodes in the cluster attempt to initiate a session with the host.

Resolution

Ensure TCP port 902 is open for all nodes within the backup cluster to permit communication with the ESXi hosts.

1. Coordinate with the Network Security Team to permit TCP port 902 between the IP addresses of all nodes in the backup cluster and the ESXi host management/vmkernel interfaces.
2. Verify that the backup node IPs are explicitly added to any "Allowed IP" lists or firewall policies on the ESXi host.
3. Test connectivity from all the backup appliances/nodes using:

   nc -zv <ESXi_IP> 902

   nping <host IP address> -p 902

4. On the ESXi host, verify if connection attempts are reaching the host by running:

   while true; do esxcli network connection list | grep :902; sleep 1; done

5. Ensure there are no subnet mask inconsistencies (e.g., 255.255.255.0 vs 255.255.0.0) between working and non-working hosts.
6. Retry the backup job once connectivity is confirmed for all nodes.