Error: "Remote host terminated the handshake" on the backup appliance during VM backup failure

Products

• VMware vCenter Server

Issue/Introduction

• Virtual machine backup jobs fail when using a backup appliance. The backup console displays the following symptoms:

• • The system returns an HTTP 500 error on backup requests.
  • The backup appliance logs contain the error: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake.
  • From the backup appliance you may also see Unable to get connection for vCenter 'vcenter'. Unable to log in to vCenter '<vCenter_FQDN>':  Post "https://vcenter:443/sdk": EOF
    
    
• The following diagram illustrates the sequence observed during the handshake failure:

           

 

Environment

VMware vCenter Server 

Cause

• Network security rules or firewall settings have Secure Sockets Layer (SSL) inspection enabled for the backup appliance access rules. This inspection interrupts the encrypted handshake between the appliance and the vCenter Server, leading to a connection termination.
  
  
• Review of the packet capture (pcap) shows the completion of the TCP three-way handshake. However, the Transport Layer Security (TLS) v1.2 Client Hello sent by the backup server does not arrive at the vCenter Server. After multiple TCP retransmissions and a 15-second idle timeout, the connection terminates with a fatal handshake failure alert and a Reset (RST) packet.

Resolution

1.To identify the issue, run a packet capture between the vCenter Server and the backup server at the same time.

a. Initiate a packet capture on the vCenter Server: tcpdump -i eth0 host [Backup_IP_ADDRESS] -s 0 -vvv -w /tmp/vcenter_failed.pcap

b. Initiate a packet capture on the backup server: tcpdump -i eth0 host [vCenter_IP_ADDRESS] -s 0 -vvv -w /tmp/backupServer_failed.pcap

c. Open the captures in Wireshark and filter for the specific Transmission Control Protocol (TCP) session port using  tcp.port == <port_number> on both files.

d. Review the complete handshake . (Refer the image on Issue / Introduction)

2. Verify the network access rules or firewall policies governing traffic between the backup appliance and the vCenter Server.

a. Disable SSL or SSH inspection on these specific access rules.

3. Perform a manual backup to verify that the appliance successfully acquires a session token.

a. Monitor the next backup cycle to confirm stable connectivity.