VAMI SFTP backup fails while creating backup directory on remote backup server

Products

• VMware vCenter Server

Issue/Introduction

• VAMI SFTP backup fails while creating backup directory on remote backup server.
• Backup log on vCenter will show similar to below entries for the failed backups :
  
  /var/log/vmware/applmgmt/backup.log

"ERROR: sftp cmd failed. RC: 2, Err: curl: (2) Failure establishing ssh session: -43, Failed getting banner"
"ERROR: Failed to create backup directory Plugin error occurred. ErrCode: 5, Args: ()""

Environment

• vCenter Server 8.x
• vCenter 9.x

Cause

For SFTP backup, vCenter tries to create a ssh tunnel for faster backup. If the tunnel creation fails, backup process fallbacks to curl commands. If the backup server has some security configurations around the number of consecutive login failures, the VC connection might get locked and backup server will not accept any further requests. So the curl commands also fail causing the backup failure.

Resolution

As SFTP backup server is blocking the connections, the backup server configurations like Fail2Ban, account lockout policies, or firewall rate-limiting needs to be reviewed to identify and white-list the triggers causing incoming requests to be blocked after consecutive authentication failures.

Workaround

For vCenter Server 8.0 U3h onwards, ssh tunnel creation for SFTP backups can be skipped by following below steps :

1. Login to vCenter Server via SSH.
2. Please take a backup of the appliance.conf file before making changes:
   
   cp /etc/applmgmt/appliance/appliance.conf /etc/applmgmt/appliance/appliance$(date '+%Y-%m-%d').conf.bak
   
   
3. Add "skipSshTunnel" configuration in the /etc/applmgmt/appliance/appliance.conf:
   
   echo "$(jq '.backupRestore.skipSshTunnel=true' /etc/applmgmt/appliance/appliance.conf )" > /etc/applmgmt/appliance/appliance.conf
   
   
4. Restart the applmgmt service :

service-control --restart applmgmt