vCenter Supervisor Backup Fails with MANIFEST_UNKNOWN Error for auto-attach-service

Products

• VMware vCenter Server

Issue/Introduction

After upgrading vCenter Server and the Supervisor to version 9.0.2, backups that include the Supervisor Control Plane fail. Manual backups that exclude the Supervisor complete successfully.

The /var/log/vmware/wcp/sv_backup_script.log file records errors fetching the auto-attach-service image, similar to the following:

DEBUG backup-service  [csp@4413] [backup.py:65] Running ['/usr/lib/vmware-wcp/tools/imgpkg', 'copy', '--bundle', 'localhost:5002/images/vksm/auto-attach-service', '--to-tar', '/var/lib/vmware/wcp/backup/xxxxx.tar']
ERROR backup-service  [csp@4413] [backup.py:72] Cmd ['/usr/lib/vmware-wcp/tools/imgpkg', 'copy', '--bundle', 'localhost:5002/images/vksm/auto-attach-service', '--to-tar', '/var/lib/vmware/wcp/backup/xxxxx.tar'] failed. ret=1, stdout=, stderr=imgpkg: Error: Fetching image: GET http://localhost:5002/v2/images/vksm/auto-attach-service/manifests/latest: MANIFEST_UNKNOWN: manifest unknown; map[Tag:latest]
Traceback (most recent call last):
File "/usr/lib/vmware-wcp/backup-restore/backup.py", line 67, in run

Cause

The svc-auto-attach-domain-xxx namespace has an incomplete image reference in its vmware-system-service-images annotation. The image URL is missing its tag, which causes the backup script to fail with a MANIFEST_UNKNOWN error when determining which images to back up.

Resolution

To workaround this issue and allow proper package updates, perform the following steps to reset the auto-attach service:

1. Log in to the VCFA UI and uninstall the auto-attach service from VKSM.

2. Reset addon status in vmsp cluster: this will allow proper update of the package yaml in VCFA

ssh vmware-system-user@$VCF_HOST
sudo su - 
kubectl patch addon broadcom.auto-attach-service-0.1.0 -n prelude --type=json \
  -p='[{"op": "remove", "path": "/status/serviceUnpackageTask"},{"op": "remove", "path": "/status/downloadTask"}]' \
  --subresource=status

3. Reinstall the auto-attach service from the VCFA UI to install it with the correct source tag.

Engineering is aware of it and working on a fix.