vCenter backup fails when the Supervisor is included.
Article ID: 401732
Updated On:
Products
Issue/Introduction
vCenter backup fails with error "BackupManager encountered an exception. See logs for details" when Supervisor is included in the backup:
/var/log/vmware/applmgmt/backup.log on vCenter:
XXXX-XX-XXTXX:XX:XX:XXX [XXXXX] [ComponentScriptsBackup:PID-2632040] [Log::run:Log.py:64] ERROR: Supervisor backup task failed. Task status (vim.TaskInfo) {
XXXX-XX-XXTXX:XX:XX:XXX [XXXXX] [ComponentScriptsBackup:PID-2632040] [Log::run:Log.py:64] ERROR: reason = 'failed to run cmd /usr/lib/vmware-wcp/backup-restore/backup.py on CPVM VirtualMachine:vm-xxxx. rc: 1, err: <nil>'
..
XXXX-XX-XXTXX:XX:XX:XXX [XXXXX] [ComponentScriptsBackup:PID-2632040] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:106] ERROR: Component backup command "/etc/vmware/backup/component-scripts/wcp/supervisors_backup_restore.py --backup" failed 1.
XXXX-XX-XXTXX:XX:XX:XXX [XXXXX] [ComponentScriptsBackup:PID-2632040] [Log::run:Log.py:64] ERROR: }Failed to take Supervisor xxxx-xxxx-xxxx-xxxx-xxxxxxxx backup: Supervisor backup task 'vim.Task:task-46267575' failedFailed to write backup content: Supervisor backup task 'vim.Task:task-46267575' failed
XXXX-XX-XXTXX:XX:XX:XXX [XXXXX] [ComponentScriptsBackup:PID-2632040] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:135] ERROR: Error during component supervisors backup
Underlying process status. rc: 1
stdout:
stderr:
Traceback (most recent call last):
"/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ComponentScripts.py", line 110, in ComponentScriptsBackup raise BackupRestoreError(('Error during component %s backup' %
util.Common.BackupRestoreError: Error during component supervisors backup
Underlying process status. rc: 1
/var/log/vmware/wcp/wcpsvc-xxxxx.log on vCenter:
XXXX-XX-XXTXX:XX:XX:XXXZ error wcp [backup/taker.go:166] [opID=backup-642b33fa-e97f-4366-b4c1-773220f1c766] Backup failed for CPVM VirtualMachine:vm-xxxx. Error: failed to run cmd
/usr/lib/vmware-wcp/backup-restore/backup.py on CPVM VirtualMachine:vm-xxxx. rc: 1, err: <nil>
XXXX-XX-XXTXX:XX:XX:XXXZ error wcp [backup/jobs.go:166] [opID=backup-642b33fa-e97f-4366-b4c1-773220f1c766] Failed to backup Supervisor xxxx-xxxx-xxxx-xxxx-xxxxxxxx. Err fail
ed to run cmd /usr/lib/vmware-wcp/backup-restore/backup.py on CPVM VirtualMachine:vm-xxxx. rc: 1, err: <nil>
XXXX-XX-XXTXX:XX:XX:XXXZ debug wcp [supervisor/task.go:127] Exit task Task:task-46267575 canceled watcher.
XXXX-XX-XXTXX:XX:XX:XXXZ info wcp [job/state.go:143] [opID=backup-642b33fa-e97f-4366-b4c1-773220f1c766] Successfully reset backup job state in DB
/var/log/vmware/wcp/supervisors_backup_restore.log on vCenter:
XXXX-XX-XX XX:XX:XX,089 __main__ ERROR - Supervisor backup task failed. Task status (vim.TaskInfo) {
error = (vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'A general system error occurred: failed to run cmd /usr/lib/vmware-wcp/backup-restore/backup.py on CPVM VirtualMachine:vm-xxxx. rc: 1, err: <nil>',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = 'failed to run cmd /usr/lib/vmware-wcp/backup-restore/backup.py on CPVM VirtualMachine:vm-xxxx. rc: 1, err: <nil>'
},
Environment
vSphere with Tanzu
Cause
There were stale replica sets with 0 desired pods in the Supervisor Cluster.
The stale replica sets could reference old images that might not be present in the registry, which in turn could cause the vCenter backup failure.
Resolution
1. SSH to the Supervisor CPVM.
2. Identify any replica sets with the desired pods value of 0.
# kubectl get rs -A
3. Delete the identified replica sets with the desired pods value of 0.
# kubectl delete rs -n <namespace> <replica_set_name>
In the event of permission issues to delete the stale replica sets, the following KB vSphere Supervisor Disk Space Clean Up Scripts can apply where a script is provided to perform the clean up required.
Feedback
