Error: "Failed to create backup directory on backup server" when performing a vCenter scheduled or manual backups
search cancel

Error: "Failed to create backup directory on backup server" when performing a vCenter scheduled or manual backups

book

Article ID: 405687

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • When a Manual or scheduled backups fails with these errors  "General system error reported by backup server" or  "Failed to create backup directory on backup server"

  • /var/log/vmware/applmgmt/backup.log reports below errors
    [MainProcess:PID-######] [BackupManager::CheckFileIntegrity:BackupManager.py:472] INFO: File integrity check is not enabled.
[MainProcess:PID-######] [SftpStorageIOLib::_addHostToKnownhosts:SftpStorageIOLib.py:221] INFO: Added remote host key to known hosts file

[MainProcess:PID-######] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts 'BackupServer_User'@'BackupServer_FQDN' echo ~

[MainProcess:PID-######] [SftpStorageIOLib::_authorizeLocalHost:SftpStorageIOLib.py:169] INFO: Authorizing localhost with command sshpass -f /tmp/tmp5ik01vrl ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts BackupServer_User@BackupServer_FQDN

[MainProcess:PID-######] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts 'BackupServer_User'@'BackupServer_FQDN' echo ~

[MainProcess:PID-######] [FtpStorageIOLib::_process_curl_output:FtpStorageIOLib.py:149] ERROR: sftp cmd failed. RC: 2, Err: curl: (2) Failure establishing ssh session: -43, Failed getting banner
, Cmd: ['/usr/bin/curl', '--fail', '-u', 'BackupServer_User:****', '--connect-timeout', '120', '--ssl-reqd', '-k', '--noproxy', '*', '--silent', '--show-error', '--list-only', 'sftp://BackupServer_FQDN/backup/scp/vmware/vcenter/']

[MainProcess:PID-######] [BackupManager::CreateTargetBackupDir:BackupManager.py:123] ERROR: Failed to create backup directory Plugin error occurred. ErrCode: 5, Args: ()

[MainProcess:PID-######] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Failed to create backup directory on backup server.

[MainProcess:PID-######] [BackupManager::main:BackupManager.py:650] INFO: skip HandleBackupCleanup as backup target directory creation failed

[MainProcess:PID-######] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk

[MainProcess:PID-######] [BackupManager::main:BackupManager.py:672] INFO: Backup job failed.

Environment

VMware vCenter Server 8.x

VMware vCenter Server 9.x

Cause

  • During SFTP backup, vCenter tries to create a passwordless ssh tunnel for faster backups.
  • This makes multiple requests to transfer keys and authorize server.
  • Based on the backup server and network configurations, this ssh tunnel creation can fail. In such cases, vCenter fallback to using curl commands.
  • However, given the many requests received earlier via SSH, curl connectivity may fail either due to firewall blocking further requests or the backup server rejecting connections from vCenter.

Resolution

To resolve this issue, force the backup process to use serial transfer mode and update the SFTP I/O library to bypass the failed tunnel initialization.

Procedure

  1. Take a snapshot of the affected vCenter Server without memory.

  2. Establish an SSH session to the vCenter Server as the root account.

  3. Stop Appliance Management Service.
    service-control --stop applmgmt
  4. Change the backup transfer mode from parallel to serial to stabilize connectivity.
    sed -i 's/parallel/serial/g' /etc/applmgmt/appliance/appliance.conf
  5. Backup existing SftpStorageIOLib.py script located in: /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/.
    mv /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py-old.bak
  6. Download the SftpStorageIOLib.py.gz script provided in attachments section of this KB  and upload it to /tmp directory if the vCenter server using any File Transfer Utility.

  7. After transferring the attached archive to the vCenter /tmp directory, decompress it
    gunzip /tmp/SftpStorageIOLib.py.gz

  8. Move the SftpStorageIOLib.py script located in /tmp to /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/

    mv /tmp/SftpStorageIOLib.py /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/

  9. Set appropriate ownership and read permissions for the python script(SftpStorageIOLib.py)

    chmod 444 /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py
chown root:root /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/SftpStorageIOLib.py
  10. Start the Appliance management service.
    service-control --start applmgmt
  11. Execute a manual backup to confirm resolution. Once successful, the scheduled backup task will also resume normal operation. Delete the snapshot once stability is confirmed

Attachments

SftpStorageIOLib.py.gz get_app
Powered by Wolken Software