VAMI backup fails when Supervisor Control Plane Nodes are included

Products

• VMware vCenter Server
• VMware vSphere Kubernetes Service

Issue/Introduction

• VAMI backup fails when Supervisor nodes are included in the backup.
• Backup is successful when Supervisor nodes are excluded.
• Upon checking the space for partitions in Supervisor Control Plane nodes with the below command, it is seen that the root partition is almost full:
  
  df -h | head

• Upon checking /var/log/vmware/applmgmt/backup.log in vCenter, below entries are seen:
  
  <date><time> [ComponentScriptsBackup:PID-######] [Log::run:Log.py:64] ERROR: Supervisor backup task failed. Task status (vim.TaskInfo)
<date><time> [ComponentScriptsBackup:PID-######] [Log::run:Log.py:64] ERROR: reason = 'failed to run cmd /usr/lib/vmware-wcp/backup-restore/backup.py on CPVM VirtualMachine:vm-<id>. rc: 1, err: <nil>'
<date><time>  [ComponentScriptsBackup:PID-#####] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:106] ERROR: Component backup command "/etc/vmware/backup/component-scripts/wcp/supervisors_backup_restore.py --backup" failed 1.

• Upon checking /var/log/vmware/wcp/wcpsvc.log , below errors are seen:
  
  <date><time> error wcp [backup/taker.go:166] [opID=backup-#### - #### - #### - ####] Backup failed for CPVM VirtualMachine:vm-<id>. Error: failed to run cmd
  /usr/lib/vmware-wcp/backup-restore/backup.py on CPVM VirtualMachine:vm-<id>. rc: 1, err: <nil>

Environment

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Cause

There is no space left on the root partition of Supervisor node for temporary files which are created during VAMI backup process.

Resolution

Perform file system cleanup in the root partition of all the Supervisor Nodes. Reference KB: vSphere Supervisor Root Disk Space Full at 100%