SDDC Backup Task fails on Subtask 'Package and Encrypt SDDC Manager Backup'

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x
• VMware Cloud Foundation 4.x

Issue/Introduction

• SDDC Backup fails at Package and Encrypt SDDC Manager Backup Stage.
• Manual and scheduled backups fail.
• Error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

  DEBUG [vcf_om,ad745799ba4f2f60,6fc2] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-7] Processing localizable exception Failed to retrieve backup workflow, invalid id
  ERROR [vcf_om,ad745799ba4f2f60,6fc2] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-7] [9Q0VF0] GET_BACKUP_FAILED_INVALID_ID Failed to retrieve backup workflow, invalid id
  com.vmware.evo.sddc.common.core.error.InvalidInputException: Failed to retrieve backup workflow, invalid id

• Error in /var/log/vmware/vcf/sddc-support/vcf-sos.log

  INFO [vcf_sos] [commandutils.py::execute_cmd_locally::206::backup-xxxxxxxx-xxxx-xxxx-xxxx-5c81e53b8263] Execute cmd: rsync -av --exclude '*.jar' --exclude 'sddc-manager-ui-app' --exclude 'lcm/tmp'  /opt/vmware/vcf /var/log/vmware/vcf/sddc-support/backup-2024-08-05-09-46-47-7669/vxxxxxx-2024-08-05-09-46-47/configuration//opt/vmware
  .
  INFO [vcf_sos] [commandutils.py::execute_cmd_locally::226::backup-xxxxxxxx-xxxx-xxxx-xxxx-5c81e53b8263] b'vcf/WF72X_VxRail-7.0.450-Composite-Upgrade-Slim-Package-for-7.0.x.zip\n'
  INFO [vcf_sos] [commandutils.py::execute_cmd_locally::226::backup-xxxxxxxx-xxxx-xxxx-xxxx-5c81e53b8263] b'vcf/java11-systemwide-java-options.conf\n'
  
  INFO [vcf_sos] [backuphelper.py::package_sddc_mgr_backup::225::backup-xxxxxxxx-xxxx-xxxx-xxxx-5c81e53b8263] Package SDDC Manager backup as a encrypted tar.gz file
  
  ERROR [vcf_sos] [commandutils.py::execute_cmd_locally::244::backup-xxxxxxxx-xxxx-xxxx-xxxx-5c81e53b8263] Std Error : ; Command cd /var/log/vmware/vcf/sddc-support/backup-2024-08-05-09-46-47-7669; tar -zcvpf xxxxxx-2024-08-05-09-46-47.tar.gz.tmp xxxx-2024-08-05-09-46-47/* did not complete even after waiting for 120 seconds

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

The 'rsync' command is unsuccessful even after waiting for 120 seconds due to large files under the directory /opt/vmware/vcf. No files (except the default files) should be kept under the /opt/vmware directory or its subsequent directories. Large files cause the timeout error. The vcf-sos.log may show clues as to which files are taking the most time during the backup operation.

Resolution

1. Check the vcf-sos.log for the backup operation using the examples in the article above and then check for files taking a long time.
2. Move any unexpected large files identified above out of the /opt/vmware/ directory or its subsequent directories.