SDDC manager backup fails with error "Package SDDC Manager backup operation failed"
search cancel

SDDC manager backup fails with error "Package SDDC Manager backup operation failed"

book

Article ID: 371850

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Backup fails on sub-task "Package and Encrypt SDDC Manager Backup" with error "Package SDDC Manager backup operation failed"


       


  • SDDC manager contains errors similar to the excerpt below:

    /var/log/vmware/vcf/sddc-support/vcf-sos.log
    ERROR [vcf_sos] [commandutils.py::execute_cmd_locally::244::backup-048f2d76-6920-4624-9019-2059839f26f1] Std Error :  ; Command cd 
/var/log/vmware/vcf/sddc-support/backup-[YYYY-MM-DDTHH:MM:SS]-2265; tar -zcvpf <FQDN>-[YYYY-MM-DDTHH:MM:SS].tar.gz.tmp 
<FQDN>-[YYYY-MM-DDTHH:MM:SS]/* did not complete even after waiting for 120 seconds
ERROR [vcf_sos] [backuphelper.py::package_sddc_mgr_backup::249::backup-048f2d76-6920-4624-9019-2059839f26f1] 
Command execution failed : cd /var/log/vmware/vcf/sddc-support/backup-[YYYY-MM-DDTHH:MM:SS]-2265; tar -zcvpf <FQDN>-[YYYY-MM-DDTHH:MM:SS].tar.gz.tmp <FQDN>-[YYYY-MM-DDTHH:MM:SS]/*
ERROR [vcf_sos] [backuphelper.py::package_sddc_mgr_backup::269::backup-048f2d76-6920-4624-9019-2059839f26f1] Package SDDC Manager backup operation failed
ERROR [vcf_sos] [backuphelper.py::package_sddc_mgr_backup::271::backup-048f2d76-6920-4624-9019-2059839f26f1] Traceback (most recent call last):
ERROR [vcf_sos] [backupservice.py::execute::78::backup-048f2d76-6920-4624-9019-2059839f26f1] Package SDDC Manager backup operation failed
ERROR [vcf_sos] [backupservice.py::execute::79::backup-048f2d76-6920-4624-9019-2059839f26f1] SDDC Manager backup operation failed in task PackageSDDCManagerBackup.

Environment

VMware SDDC Manager 5.x

Cause

This issue can occur if the root or the nfs-mount partition of the SDDC manager are running low on free space. 

Scenario 1: Root partition has a very large file, such as an old log bundle. The backup process, which expects smaller configuration files, may time out after 120 seconds when attempting to compact very large files.

Scenario 2: The storage exhaustion on the SDDC Manager's internal NFS mount (> 90%). Before a backup is transferred to an external SFTP destination, the SDDC Manager must package, compress, and encrypt the configuration files locally. If the system lacks sufficient temporary "scratch space" on the NFS partition, the packaging task fails.

 

Resolution

Validate the utilization details of the SDDC manager: 

df -h

Scenario 1: Root low on space

The top 20 files consuming the most disk space need to be removed.

  • Use the following command within the root directory to identify the top 20 files with high disk usage:

    du -a <Path> | sort -n -r | head -n 20 

Please contact Broadcom support for assistance in identifying and safely deleting these files.

Scenario 2: NFS-mount low on space

  1. Take a snapshot of the SDDC Manager VM.
  2. Run the Bundle Cleanup Utility to systematically purge old or already installed bundles. This utility is required to safely remove files from both the disk and the internal PostgreSQL inventory database without breaking dependencies:

 

Additional Information

Possible reasons for orphaned backup files left on the SDDC manager

  • The backup packaging step (tar + encrypt) has a 2-minute timeout. The backup dataset could grow to large file size, and packaging alone now takes longer than 2 minutes. The process gets forcibly terminated mid-encryption every time, causing the backup to fail at the same step consistently.

 

Backup files cleanup process (2 day waiting period)

  • No backup files are silently accumulated from failed attempts; failed backup tasks do not leave orphaned files behind.
  • Only incomplete/orphaned backups get auto-cleaned, only after 2 days, and only the next time a backup successfully runs (not on a fixed schedule).
Powered by Wolken Software