Scheduled vCenter Backups Fail with "Failed getting banner" Error in 9.0.x.

Products

• VMware vCenter Server
• VCF Operations

Issue/Introduction

 

• The issue occurs specifically when the backup configuration is set to retain only the most recent X backups.
• Scheduled backups consistently fail at 95% during the cleanup stage. Manual backups, however, complete successfully regardless of the configured retention policy. Scheduled backups also complete successfully when the “Retain All Backups” option is enabled.
• This behavior indicates that the problem is likely related to the backup retention mechanism responsible for removing older backup files from the remote storage destination.
• Under vCenter's /var/log/vmware/applmgmt/backup.log, below error logs are observed:
  
  [MainProcess:PID-1547####] [FtpStorageIOLib::_process_curl_output:FtpStorageIOLib.py:149] ERROR: sftp cmd failed. RC: 2, Err: curl: (2) Failure establishing ssh session: -13, Failed getting banner

 

 

Environment

• VMware Cloud Foundation (VCF) Operations 9.0.x
• vCenter 9.0.x

Cause

• Configuring scheduled backups with a specific retention count "Retain last X Backups" is currently triggering backup failures.

Resolution

Since the curl commands used to remove the backup files were failing with the "Failed getting banner" error, we need to update FtpStorageIOLib.py to add retry handling for curl command calls on the vCenter.

To resolve this issue, follow the steps below:

1. Take a Snapshot of the vCenter.
2. SSH to the vCenter Server as root.
3. Backup the existing config file:
   
   cp /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py.backup
   
   
4. Open the file vi editor:
   
   vi /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py
   
   
5. Locate the _run_cmd_retries function (around line 26). Find these lines:
   
   if cmd_res.returncode == 28:
raise PluginError(ErrCodes.timeout)
return cmd_res
   
   
6. Insert two lines immediately after raise PluginError(ErrCodes.timeout) so it reads:
   
   if cmd_res.returncode == 28:
raise PluginError(ErrCodes.timeout)
if cmd_res.returncode == 2:
raise PluginError(ErrCodes.connection_err)
return cmd_res
   
   
7. Press Esc wq! to save and exit vi editor.
8. Restart the appliance management service:
   
   service-control --restart applmgmt