/storage/lifecycle/vmware-wcp/backup/archives directory on vCenter Server keeps filling up due to incomplete vSphere Supervisor Backups

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Disk space within the vCenter Server is being consumed rapidly as the /storage/lifecycle/vmware-wcp/backup/archives directory fills up. The directory contains multiple large backup files that are incomplete or abandoned. 
• This accumulation coincides with failed vCenter Server backup jobs when the Supervisor Cluster is included in the backup configuration.
• Analysis of the vCenter Server backup logs at /var/log/vmware/applmgmt/backup.log confirms that backup processes are failing intermittently:

  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is -13, but expected exit codes are [0].
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr: Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
      main(sys.argv[1], sys.argv[2], sys.argv[3])
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
      stdout_obj.write(data)
  BrokenPipeError: [Errno 32] Broken pipe
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Proc::GetProcsStatus:Proc.py:333] INFO: Skip to report the error.
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is 1, but expected exit codes are [0].
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Proc::GetProcsStatus:Proc.py:361] ERROR: rc: 55, stderr: b'curl: (55) Send failure: Broken pipe\n'
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Proc::UpdateExceptionStatus:Proc.py:384] ERROR: Checksum not generated at /dev/shm/backupRestoreSumFile-<File_Name>
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Log::run:Log.py:64] ERROR: Failed to write backup content: [Errno 32] Broken pipeException ignored in: <_io.TextIOWrapp
  er name='<stdout>' mode='w' encoding='utf-8'>
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [Log::run:Log.py:64] ERROR: BrokenPipeError: [Errno 32] Broken pipe
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:106] ERROR: Component backup command "/etc/vmware/backup/component-scripts/wcp/supervisors_backup_restore.py --backup" failed 120.
  YYYY-MM-DDTHH:MM:SS [ComponentScriptsBackup:PID-######] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:135] ERROR: Error during component supervisors backup

Environment

VMware vCenter Server

Cause

This issue is caused by the behavior of the Supervisor Cluster backup mechanism during a failure event. When the main vCenter Server backup process fails intermittently while the Supervisor Cluster is configured to be included, the failure triggers a fallback or an ad-hoc backup of the Supervisor Cluster which is the stored at the following directory: /storage/lifecycle/vmware-wcp/backup/archives

Resolution

To resolve the issue permanently, the root cause of the vCenter Server's backup failure needs to be determined and fixed. 
As a workaround, files within the /storage/lifecycle/vmware-wcp/backup/archives directory can be removed and purged by following the below steps:  

NOTE: Take a backup of the vCenter Server.

• Open an SSH session to vCenter Server. 
• Run the following commands: 
  • cd /storage/lifecycle/vmware-wcp/backup/archives
  • pwd (This is to ensure that present working directory is /storage/lifecycle/vmware-wcp/backup/archives)
  • rm -rf *
• Run the following command to confirm if disk space is reduced: 
  • df -h

Additional Information

vCenter backup fails when the Supervisor is included.

vCenter backup fails when including Supervisor cluster

VAMI backups fail intermittently when Supervisor Control Plane is included.