Error: "Failed to create backup directory on backup server" when performing a vCenter scheduled or manual backups

Products

• VMware vCenter Server

Issue/Introduction

• When a Manual or scheduled backups fails with these errors  "General system error reported by backup server" or  "Failed to create backup directory on backup server"

• /var/log/vmware/applmgmt/backup.log reports below errors

  [MainProcess:PID-######] [BackupManager::CheckFileIntegrity:BackupManager.py:472] INFO: File integrity check is not enabled.
  [MainProcess:PID-######] [SftpStorageIOLib::_addHostToKnownhosts:SftpStorageIOLib.py:221] INFO: Added remote host key to known hosts file
  
  [MainProcess:PID-######] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts 'BackupServer_User'@'BackupServer_FQDN' echo ~
  
  [MainProcess:PID-######] [SftpStorageIOLib::_authorizeLocalHost:SftpStorageIOLib.py:169] INFO: Authorizing localhost with command sshpass -f /tmp/tmp5ik01vrl ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts BackupServer_User@BackupServer_FQDN
  
  [MainProcess:PID-######] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts 'BackupServer_User'@'BackupServer_FQDN' echo ~
  
  [MainProcess:PID-######] [FtpStorageIOLib::_process_curl_output:FtpStorageIOLib.py:149] ERROR: sftp cmd failed. RC: 2, Err: curl: (2) Failure establishing ssh session: -43, Failed getting banner
  , Cmd: ['/usr/bin/curl', '--fail', '-u', 'BackupServer_User:****', '--connect-timeout', '120', '--ssl-reqd', '-k', '--noproxy', '*', '--silent', '--show-error', '--list-only', 'sftp://BackupServer_FQDN/backup/scp/vmware/vcenter/']
  
  [MainProcess:PID-######] [BackupManager::CreateTargetBackupDir:BackupManager.py:123] ERROR: Failed to create backup directory Plugin error occurred. ErrCode: 5, Args: ()
  
  [MainProcess:PID-######] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Failed to create backup directory on backup server.
  
  [MainProcess:PID-######] [BackupManager::main:BackupManager.py:650] INFO: skip HandleBackupCleanup as backup target directory creation failed
  
  [MainProcess:PID-######] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
  
  [MainProcess:PID-######] [BackupManager::main:BackupManager.py:672] INFO: Backup job failed.

Environment

VMware vCenter Server 8.x

VMware vCenter Server 9.x

Cause

• During SFTP backup, vCenter tries to create a passwordless ssh tunnel for faster backups.
• This makes multiple requests to transfer keys and authorize server.
• Based on the backup server and network configurations, this ssh tunnel creation can fail. In such cases, vCenter fallback to using curl commands.
• However, given the many requests received earlier via SSH, curl connectivity may fail either due to firewall blocking further requests or the backup server rejecting connections from vCenter.

Resolution

NOTE: If you are using vCenter 8.0u3k or higher, please use the resolution in KB 366953 as the script will no longer be compatible.

To resolve this issue, force the backup process to use serial transfer mode and update the SFTP I/O library to bypass the failed tunnel initialization.

Procedure

1. Take a snapshot of the affected vCenter Server without memory.

2. Establish an SSH session to the vCenter Server as the root account.

3. Stop the Appliance Management Service.

   service-control --stop applmgmt

4. Change the backup transfer mode from parallel to serial to stabilize connectivity.

   sed -i 's/parallel/serial/g' /etc/applmgmt/appliance/appliance.conf

5. Back up the existing SftpStorageIOLib.py script located in /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/:

   mv /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py-old.bak

6. Download the SftpStorageIOLib.py.gz script provided in the Attachments section of this KB and upload it to the /tmp directory of the vCenter Server using any file transfer utility.

7. After transferring the attached archive to the vCenter /tmp directory, decompress it:

   gunzip /tmp/SftpStorageIOLib.py.gz

8. Move the SftpStorageIOLib.py script located in /tmp to /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/:

   mv /tmp/SftpStorageIOLib.py /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/

9. Set appropriate ownership and read permissions for the Python script (SftpStorageIOLib.py):

   chmod 444 /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py
   chown root:root /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py

10. Start the Appliance Management Service.

    service-control --start applmgmt

11. Execute a manual backup to confirm resolution. Once successful, the scheduled backup task will also resume normal operation. Delete the snapshot once stability is confirmed.