Scheduled SFTP VAMI backups fail intermittently on vCenter when using Windows SFTP server

Products

• VMware vCenter Server

Issue/Introduction

Scheduled SFTP backups for the vCenter Server Appliance (vCSA) fail intermittently or are skipped entirely. Key indicators include:

• The backup job is missing from the list of backups in the VAMI (vCenter Appliance Management Interface).
• No failure is immediately visible in the VAMI UI.
• Scheduled backups appear to be skipped.
• /var/log/vmware/applmgmt/backupScheduler.log in vCenter shows below entries:
  
  <date><time> [0] [MainProcess:PID-1651699] [SchedulerCron::main:SchedulerCron.py:129] INFO: Issuing the scheduled backup request for schedule: daily.
<date><time> [0] [MainProcess:PID-1651699] [Scheduler::ExecScheduleRun:Scheduler.py:137] ERROR: Failed to issue the Schedules.run request. Exception: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1395, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 323, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 292, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response


• During the similar time when the above error is seen, applmgmt-health API times out. This can be validated in /var/log/vmware/applmgmt/appliance-health.log in vCenter:
  
  <date><time> [1652511]ERROR:applmgmt-health:ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=8201): Read timed out. (read timeout=5)"))
<date><time> [1652511]INFO:applmgmt-health:Retry count: 2
<date><time> [1652511]ERROR:applmgmt-health:ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=8201): Read timed out. (read timeout=5)"))
<date><time> [1652511]INFO:applmgmt-health:Retry count: 3
<date><time> [1652511]ERROR:applmgmt-health:ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=8201): Read timed out. (read timeout=5)"))
<date><time> [1653347]INFO:applmgmt-health:Getting applmgmt health...
<date><time> [1653347]INFO:applmgmt-health:Args: Timeout=5, Retry=3


• Due to this health API failure, vmon finds applmgmt unhealthy and restarts the applmgmt service during the same time. Below entries can be seen in /var/log/vmware/vmon/vmon.log in vCenter:
  
  <date><time> Wa(03) host-2373 <applmgmt> Health of service failed. Health data:
<date><time>In(05) host-2373 <applmgmt> Recover from service api health check failure. Fail count 0
<date><time> In(05) host-2373 <applmgmt> Restarting service.
<date><time> Wa(03) host-2373 <applmgmt> Found empty StopSignal parameter in config file. Defaulting to SIGTERM
<date><time> In(05) host-2373 <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata applmgmt,UNHEALTHY,HEALTHY,1
<date><time> Wa(03) host-2373 <applmgmt> Sysprocess clean stop timed out. Force kill. Pid 3526295
<date><time> Wa(03) host-2373 <applmgmt> Service exited. Exit code 1
<date><time> In(05) host-2373 <applmgmt> Service STARTED successfully

Environment

vCenter Server 8.x

Cause

This issue occurs because the applmgmt service attempts to establish a passwordless SSH connection using ssh-copy-id before every SFTP backup.

On Windows-based SFTP servers, this operation fails because there is no Unix shell to write the authorized_keys file. The service then waits for the SSH connection to time out, which can take more than 60 seconds. During this timeout period:

• The applmgmt service remains busy and unresponsive to health checks.
• The applmgmt-health API fails after multiple retries.
• The vmon service detects the health failure and restarts the applmgmt service.
• The restart terminates the active backup job, causing it to fail or never start.
• Below entries can be seen in /var/log/vmware/backupScheduler.log in vCenter where ssh-copy-id fails and falls back to Curl. If this operation takes more than 60 seconds, this issue can occur:
  
  <date><time> [SftpStorageIOLib::_authorizeLocalHost] WARNING: Failed to authorize local host with command: sshpass ... ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 ... <backup_user>@<backup_server>. Will proceed with backup through curl
<date><time> [MainProcess:PID-1112438][SftpStorageIOLib::_setupPasswordlessTunnel:SftpStorageIOLib.py:141] WARNING: Passwordless ssh tunnel is not working, falling back to CURL

Resolution

Broadcom Engineering is aware about this issue and this will be fixed in a future release.

Workaround:

Configure the vCenter Server Appliance to bypass the SSH tunnel creation and proceed directly to the file transfer using curl. Follow the below steps:

1. Log in to the vCenter Server Appliance via SSH as the root user.
2. Create a backup of the existing appliance configuration file:
   
   cp /etc/applmgmt/appliance/appliance.conf /etc/applmgmt/appliance/appliance.conf.bak
   
   
3. Use jq to add the skipSshTunnel parameter and set it to true:
   
   echo "$(jq '.backupRestore.skipSshTunnel=true' /etc/applmgmt/appliance/appliance.conf )" > /etc/applmgmt/appliance/appliance.conf
   
   
4. Restart the appliance management service for the changes to take effect:
   
   service-control --restart applmgmt