VAMI backup failing with SFTP fails with error:"Failure establishing ssh session: -43, Failed getting banner"

Products

• VMware vCenter Server

Issue/Introduction

• /var/log/vmware/applmgmt/backup.log reports below errors

YYYY-MM-DDTHH:MM:06.306 [20250429-092002-24322831] [MainProcess:PID-925041] [SftpStorageIOLib::_authorizeLocalHost:SftpStorageIOLib.py:169] INFO: Authorizing localhost with command sshpass -f /tmp/tmp5ik01vrl ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts backup_username@backup_server_fqdn
YYYY-MM-DDTHH:MM:07.184 [20250429-092002-24322831] [MainProcess:PID-925041] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts backup_username@backup_server_fqdn' echo ~
YYYY-MM-DDTHH:MM:09.730 [20250429-092002-24322831] [MainProcess:PID-925041] [FtpStorageIOLib::_process_curl_output:FtpStorageIOLib.py:149] ERROR: sftp cmd failed. RC: 2, Err: curl: (2) Failure establishing ssh session: -43, Failed getting banner
, Cmd: ['/usr/bin/curl', '--fail', '-u', 'svc-vmware-backup:****', '--connect-timeout', '120', '--ssl-reqd', '-k', '--noproxy', '*', '--silent', '--show-error', '--list-only', 'sftp://backup_server_fqdn/backup/scp/vmware/vcenter/']
YYYY-MM-DDTHH:MM:09.730 [20250429-092002-24322831] [MainProcess:PID-925041] [BackupManager::CreateTargetBackupDir:BackupManager.py:123] ERROR: Failed to create backup directory Plugin error occurred. ErrCode: 5, Args: ()
YYYY-MM-DDTHH:MM:09.731 [20250429-092002-24322831] [MainProcess:PID-925041] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Failed to create backup directory on backup server.
YYYY-MM-DDTHH:MM:09.731 [20250429-092002-24322831] [MainProcess:PID-925041] [BackupManager::main:BackupManager.py:650] INFO: skip HandleBackupCleanup as backup target directory creation failed

• This backup failure was observed after RHEL 9(backup server) version upgrade to 9
• Passwordless tunnel creation fails

Environment

vCenter server 8.0.3

Cause

The issue occurs when the SSH passwordless tunnel creation fails, and the backup proceeds using curl.
There might be some security or firewall configuration on the backup server that is rejecting the vCenter connection due to consecutive login failures during the passwordless tunnel setup.

Resolution

This is a known issue related to the backup server. In a future release, Broadcom Engineering is expected to provide a workaround that involves adding specific settings to the appliance.conf file to bypass passwordless tunnel creation. However, the root cause of the issue still needs to be investigated on the backup server side.