'Backup Manager encountered an exception' error during vCenter VAMI Backup using SFTP

Products

VMware vCenter Server

Issue/Introduction

SFTP backup fails during data transfer with an error "Backup Manager encountered an exception. See logs for details".

The following errors are logged in the /var/log/vmware/applmgmt/backup.log file:

ERROR: rc: 120, stderr: Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

Err: curl: (2) Failure establishing ssh session: -43, Failed getting banner

The VAMI backup log, /var/log/vmware/applmgmt/backup.log, might have entries similar to the following ones:

[MainProcess:PID-###] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:160] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts -o User=<username> <backup_server_fqdn_or_ip> -o PreferredAuthentications=publickey echo ~
[MainProcess:PID-###] [SftpStorageIOLib::_setupPasswordlessTunnel:SftpStorageIOLib.py:141] WARNING: Passwordless ssh tunnel is not working, falling back to CURL
[MainProcess:PID-###] [FtpStorageIOLib::_process_curl_output:FtpStorageIOLib.py:152] ERROR: sftp cmd failed. RC: 2, Err: curl: (2) Failure establishing ssh session: -43, Failed getting banne, Cmd: ['/usr/bin/curl', '--fail', '-u', '<username>:****', '--connect-timeout', '120', '--ssl-reqd', '-k', '--noproxy', '*', '--silent', '--show-error', '--list-only', 'sftp://<backup_server_fqdn_or_ip>/<folder>/<vCenter_fqdn>/vCenter/<folder>/M_###/'], stdout:
[MainProcess:PID-###] [BackupManager::CreateTargetBackupDir:BackupManager.py:123] ERROR: Failed to create backup directory Plugin error occurred. ErrCode: 5, Args: ()
[MainProcess:PID-###] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Failed to create backup directory on backup server.

If the SFTP backup server uses OpenSSH 9.8 or later, its syslog or messages log (usually found in /var/log/, but the location depends on the operating system) might contain messages similar to:

MM DD hh:mm:ss sftp-server sshd[###]: drop connection #0 from <vCenter_IP>:<port> on <sftp_server_ip>:22 penalty: failed authentication
MM DD hh:mm:ss sftp-server sshd[###]: drop connection #0 from <vCenter_IP>:<port> on <sftp_server_ip>:22 penalty: failed authentication
MM DD hh:mm:ss sftp-server sshd[###]: drop connection #0 from <vCenter_IP>:<port> on <sftp_server_ip>:22 penalty: failed authentication
MM DD hh:mm:ss sftp-server sshd[###]: drop connection #0 from <vCenter_IP>:<port> on <sftp_server_ip>:22 penalty: failed authentication

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue can be caused by a limitation of concurrent SSH Sessions on the SFTP Server.
In its default configuration the VAMI backup component tries to create multiple parallel streams with the backup server, in order to achieve optimal backup performance. The number of these session is automatically determined.
If the number of sessions vCenter tries to open exceeds the session limitation of the backup server, the functionality of the SFTP backup can be negatively impacted, causing the backup tasks to fail.

Additionally, current vCenter version will initially attempt to open the sessions with the backup server using its internal host keys due to a limitation in OpenSSH, before connecting with the username and password configured in the backup task. However, in most scenarios these keys are not trusted by the backup server.
If the backup server uses OpenSSH 9.8 or a later version, this might trigger the OpenSSH Penalty feature, which blocks SSH/FTP/SFTP clients after a preconfigured number of unsuccessful authentication attempts.

Resolution

VMware engineering is aware of this and are working on on implementing a fix in a future vCenter version.

In the meantime there are 2 options to work around the issue:

Option A - Switch the backup file transfer to serial instead of parallel mode

To switch the file transfer mode, apply the following steps:

Open an SSH connection to the vCenter Server Appliance (VCSA) and login as root

Create a backup copy of the file /etc/applmgmt/appliance/appliance.conf:

# cp /etc/applmgmt/appliance/appliance.conf /storage/appliance.conf.bak

Edit the file:

# vi /etc/applmgmt/appliance/appliance.conf

Change the following line:

"fileTransferMode": "parallel"

to

"fileTransferMode": "serial"

Save the file by pressing ESC and entering the following command in the VI editor:
```
:wq!
```
Restart the appliance management service (applmgmt):
```
# service-control --restart applmgmt
```

Option B - Disable SSH tunnel creation

In order to reduce the time needed to write the backup to the SFTP server, vCenter tries to create an SSH tunnel. If the tunnel creation fails, the backup process falls back to using curl. In order to prevent the unsuccessful tunnel attempt from adding to the count for the penalty calculation, starting with vCenter Server 8.0 Update 3h, the tunnel feature can be disabled, using the following steps:

Open an SSH connection to the vCenter Server Appliance (VCSA) and login as root

Create a backup copy of the file /etc/applmgmt/appliance/appliance.conf:

# cp /etc/applmgmt/appliance/appliance.conf /storage/appliance.conf.bak

Run the following command to add the "skipSshTunnel" option to the file as true:

# echo "$(jq '.backupRestore.skipSshTunnel=true' /etc/applmgmt/appliance/appliance.conf )" > /etc/applmgmt/appliance/appliance.conf

Restart the appliance management service (applmgmt):
```
# service-control --restart applmgmt
```

Note: In many situations disabling public key authentication and limit the authentication methods available on the SFTP backup server to only user/password might be sufficient to work around the issue. Refer to vCenter server filebased backups are failing for the SFTP protocol for more information.