"curl: (56) Recv failure: Connection reset by peer" Error When vCenter Server Performs FTP Backups

Products

• VMware vCenter Server

Issue/Introduction

When attempting to perform a vCenter Server backup to an FTP server,

• The backup job fails and shows a "Failed" status in the vCenter Server Appliance Management Interface (VAMI)
• No success confirmation is received
• The backup files are not created on the FTP server
• The backup history in the VAMI shows repeated failures
• No successful backup files are available for recovery purposes
• Examining the backup logs located at /var/log/vmware/applmgmt/backup.log reveals detailed error messages including:
  • curl: (56) Recv failure: Connection reset by peer
    • This failure occurs consistently across all backup components during data transfer.
  • /usr/bin/tar: -: Wrote only 8192 of 10240 bytes
  • Error is not recoverable: exiting now

Environment

• vCenter Server (any version with built-in backup functionality)
• Backup method: FTP protocol
• Target environment: External FTP server
• Affected backup components: All components (StellarDB, Lotus DB, VCDB, VTSDB, ConfigFiles)

Cause

The "Connection reset by peer" error occurs when the network connection between the vCenter Server and the FTP server is unexpectedly terminated during data transfer. This can happen for several reasons:

1. Network connectivity issues between vCenter Server and the FTP server
2. Firewall or network device interrupting established connections
3. FTP server resource limitations (CPU, memory, or bandwidth constraints)
4. Insufficient timeout settings for large data transfers
5. Authentication issues during longer sessions
6. Limited storage space on the FTP server causing connection termination

Resolution

Follow these steps to troubleshoot and resolve vCenter Server FTP backup failures:

1. Verify basic network connectivity between vCenter Server and the FTP server:

   1. From the vCenter Server Appliance shell, use ping to test basic connectivity to the FTP server

   2. Check if the FTP port (typically port 21) is accessible using netcat where you substitute the actual target FTP server hostname for <hostname>::
      
      nc -zv ftp_server_<hostname> 21

2. Check FTP server connection using manual FTP commands:

   1. From the vCenter Server Appliance shell, connect to the FTP server manually where you substitute the actual target FTP server hostname for <hostname>:
      
      ftp ftp_server_<hostname>

   2. Attempt to upload a small test file to verify write permissions

   3. Attempt to upload a larger test file (100MB+) to verify stability during longer transfers

3. Review and adjust FTP timeout settings:

   1. In the vCenter Server Appliance Management Interface (VAMI), navigate to Backup > Configure

   2. If available, increase the connection timeout settings for the backup job

   3. On the FTP server, increase idle timeout settings if possible

4. Examine network infrastructure:

   1. Check if any network devices (firewalls, load balancers) have idle timeout settings that might disconnect long-running FTP sessions

   2. Verify that no bandwidth throttling is active that could cause slow transfers

   3. Ensure that stateful inspection on firewalls isn't terminating the FTP control or data connections

5. Consider alternative backup protocols:

   1. If available, switch to a more reliable protocol like FTPS, SCP, or HTTP/HTTPS

   2. Configure the new protocol in the vCenter Server Appliance Management Interface (VAMI)

   3. Test a backup using the new protocol

6. Check storage capacity and permissions:

   1. Verify sufficient disk space exists on the FTP server

   2. Ensure the FTP user has proper write permissions to the destination directory

   3. Check if any quotas are enabled that might restrict upload size

Additional Information

• For very large vCenter deployments, consider increasing the maximum transfer unit (MTU) to improve network performance if your infrastructure supports jumbo frames
• Related documentation:
  • File-Based Backup and Restore of vCenter Server
  • Backup and Restore Protocols for vCenter Server