VAMI backups fail intermittently when Supervisor Control Plane is included.

Products

• VMware vCenter Server
• VMware vSphere Kubernetes Service

Issue/Introduction

• vCenter Server backups are successful without Supervisor Control Plane but fail intermittently when "Supervisor Control Plane" option is selected.
• Individual Supervisor backups are successful when performed manually.
• Below log snippets are observed in /var/log/vmware/applmgmt/backup.log

YYYY-MM-DDTHH:MM:SS [190008-24674346] [ComponentScriptsBackup:PID-314574] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:88] INFO: Execute supervisors script: /etc/vmware/backup/component-scripts/wcp/supervisors_backup_restore.py --backup
YYYY-MM-DDTHH:MM:SS [190008-24674346] [ComponentScriptsBackup:PID-314574] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:95] INFO: Dispatching stream.
YYYY-MM-DDTHH:MM:SS [190008-24674346] [ComponentScriptsBackup:PID-314574] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is -13, but expected exit codes are [0].
YYYY-MM-DDTHH:MM:SS [190008-24674346] [ComponentScriptsBackup:PID-314574] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr: Traceback (most recent call last):
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
main(sys.argv[1], sys.argv[2], sys.argv[3])
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
stdout_obj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

• Backup terminates due to "Error in the SSH layer".

YYYY-MM-DDTHH:MM:SS [190008-24674346] [ComponentScriptsBackup:PID-314574] [Proc::GetProcsStatus:Proc.py:333] INFO: Skip to report the error.
YYYY-MM-DDTHH:MM:SS [190008-24674346] [ComponentScriptsBackup:PID-314574] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is 1, but expected exit codes are [0].
YYYY-MM-DDTHH:MM:SS [190008-24674346] [ComponentScriptsBackup:PID-314574] [Proc::GetProcsStatus:Proc.py:361] ERROR: rc: 79, stderr: b'curl: (79) Error in the SSH layer\n'

• Backup process returns error code -13.

YYYY-MM-DDTHH:MM:SS [164635-24322831] [ComponentScriptsBackup:PID-239050] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is -13, but expected exit codes are [0].

Environment

VMware vSphere with Tanzu

VMware vCenter Server

Cause

Short keep alive / broken connection timeout on SFTP server causes the backup to fail intermittently.

Resolution

Modify the SFTP settings for backup server to increase the keep alive / broken connection timeout.