vCenter scheduled SFTP backups fail intermittently with curl (28) timeout when using a Windows SFTP server

Products

• VMware vCenter Server

Issue/Introduction

Scheduled file-based backups for the vCenter Server Appliance (vCSA) fail intermittently. Observed symptoms include:

• Backup jobs appear in an Error state in the VAMI (vCenter Appliance Management Interface).
• The failure occurs at different points in the backup sequence each time.
• Manual backups may succeed if performed outside the problematic window.
• In /var/log/vmware/applmgmt/backup.log on the vCenter Server:
  
  

  <date><time> [MainProcess:PID-260325] [BackupManager::main:BackupManager.py:618] INFO: Starting backup job...
<date><time> [MainProcess:PID-260325] [BackupManager::main:BackupManager.py:625] INFO: Scheduled Backup triggerd
<date><time> [LotusBackup:PID-263682] [Proc::GetProcsStatus:Proc.py:383] ERROR: rc: 28, stderr: b'curl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\ncurl: (28) Timeout was reached\n'

• In the C:\ProgramData\SSH\logs\sshd.log on the Windows SFTP server:
  

  <date><time> error: lookup_principal_name: User principal name lookup failed for user '<backup_username>' (explicit: 1722, implicit: 1722)
<date><time> fatal: ga_init, unable to resolve user <backup_username>

Environment

vCenter Server 8.x

Cause

This issue occurs when a Windows-based SFTP server (such as Win32-OpenSSH) uses a domain account for authentication.

Intermittently, the Windows SFTP server loses connectivity to the Domain Controller (DC) while trying to resolve the user principal. This triggers a Windows RPC error 1722 (RPC_S_SERVER_UNAVAILABLE). Because the server hangs while waiting for the DC response, the vCenter curl session receives no data and eventually times out.

Resolution

To resolve this issue, follow either of the below options:

• Change the SFTP backup user to a local Windows account on the Backup Server instead of a domain account. Local authentication has no DC dependency and removes this entire failure class
• Investigate with your Network or Active Directory team to find why the SFTP server is intermittently losing DC connectivity and remediate the issue.