When performing a vCenter backup via VAMI over SFTP, the process fails with the error: "Failed to create backup directory on backup server"

Products

• VMware vCenter Server

Issue/Introduction

• When a Manual or scheduled backups fails with these errors  "General system error reported by backup server" or  "Failed to create backup directory on backup server"

• /var/log/vmware/applmgmt/backup.log reports below errors

YYYY-MM-DDTHH:MM:SS  [MainProcess:PID-######] [BackupManager::CheckFileIntegrity:BackupManager.py:472] INFO: File integrity check is not enabled.
YYYY-MM-DDTHH:MM:SS  [MainProcess:PID-######] [SftpStorageIOLib::_addHostToKnownhosts:SftpStorageIOLib.py:221] INFO: Added remote host key to known hosts file
YYYY-MM-DDTHH:MM:SS  [MainProcess:PID-######] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts 'BackupServer_User'@'BackupServer_FQDN' echo ~
YYYY-MM-DDTHH:MM:SS  [MainProcess:PID-######] [SftpStorageIOLib::_authorizeLocalHost:SftpStorageIOLib.py:169] INFO: Authorizing localhost with command sshpass -f /tmp/tmp5ik01vrl ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts BackupServer_User@BackupServer_FQDN
YYYY-MM-DDTHH:MM:SS  [MainProcess:PID-######] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts 'BackupServer_User'@'BackupServer_FQDN' echo ~
YYYY-MM-DDTHH:MM:SS  [MainProcess:PID-######] [FtpStorageIOLib::_process_curl_output:FtpStorageIOLib.py:149] ERROR: sftp cmd failed. RC: 2, Err: curl: (2) Failure establishing ssh session: -43, Failed getting banner
, Cmd: ['/usr/bin/curl', '--fail', '-u', 'BackupServer_User:****', '--connect-timeout', '120', '--ssl-reqd', '-k', '--noproxy', '*', '--silent', '--show-error', '--list-only', 'sftp://BackupServer_FQDN/backup/scp/vmware/vcenter/']
YYYY-MM-DDTHH:MM:SS:09.730  [MainProcess:PID-######] [BackupManager::CreateTargetBackupDir:BackupManager.py:123] ERROR: Failed to create backup directory Plugin error occurred. ErrCode: 5, Args: ()
YYYY-MM-DDTHH:MM:SS:09.731  [MainProcess:PID-######] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Failed to create backup directory on backup server.
YYYY-MM-DDTHH:MM:SS:09.731  [MainProcess:PID-######] [BackupManager::main:BackupManager.py:650] INFO: skip HandleBackupCleanup as backup target directory creation failed
YYYY-MM-DDTHH:MM:SS:09.967  [MainProcess:PID-######] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
YYYY-MM-DDTHH:MM:SS:09.972  [MainProcess:PID-######] [BackupManager::main:BackupManager.py:672] INFO: Backup job failed.

Environment

vCenter Server 8.0.3

Cause

During SFTP backup, vCenter tries to create a passwordless ssh tunnel for faster backups. This makes multiple requests to transfer keys and authorize server. But based on the backup server and network configurations, this ssh tunnel creation can fail. In such cases, vCenter fallback to using curl commands.
However, given the many requests received earlier via SSH, curl connectivity may fail either due to firewall blocking further requests or the backup server rejecting connections from vCenter.

Resolution

As the issue is environmental, the below workaround can be attempted to bypass passwordless SSH and use curl directly during SFTP backups.

Verify curl connectivity from vCenter to the backup server and confirm that the backup directory contents are returned.
/usr/bin/curl --fail -u BackupServer_User:BackupServer_Pasword --connect-timeout 120 --ssl-reqd -k --noproxy '*' --silent --show-error --list-only 'sftp://BackupServer_FQDN/backup_directory/'

Then, apply the below workaround:

1) Stop applmgmt service.
service-control --stop applmgmt

2) Change the backup transfer mode to serial to prevent multiple transfers at the same time.
sed -i 's/parallel/serial/g' /etc/applmgmt/appliance/appliance.conf

3) Download the attached SftpStorageIOLib.py file and replace the existing file in the path shown below.
/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py

4) Assign the appropriate ownership and permissions to the updated file.
chmod 444 /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py
chown root:root /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py

5) Start applmgmt service
service-control --start applmgmt