SDDC backups fail due to file copy timeouts

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Scheduled backups fail at the 'Backup SDDC Manager Configuration' stage
• Manual backups fail at same stage
• sos logs contain errors similar to below:

2022-10-17T08:14:39.481Z [INFO commandutils.py::execute_cmd_locally::226::backup-39eaeb5e-bd78-459e-988f-098c6ab9f91c] b'vcf/OSL/open_source_license.txt\n'
2022-10-17T08:14:39.486Z [INFO commandutils.py::execute_cmd_locally::226::backup-39eaeb5e-bd78-459e-988f-098c6ab9f91c] b'vcf/SoSLogs/\n'
2022-10-17T08:14:39.489Z [INFO commandutils.py::execute_cmd_locally::226::backup-39eaeb5e-bd78-459e-988f-098c6ab9f91c] b'vcf/SoSLogs/sos-2022-10-06-10-31-36-
3699.tar.gz\n'
2022-10-17T08:16:39.660Z [ERROR commandutils.py::execute_cmd_locally::244::backup-39eaeb5e-bd78-459e-988f-098c6ab9f91c] Std Error :  ; Command rsync -av --ex
clude '*.jar' --exclude 'sddc-manager-ui-app' --exclude 'lcm/tmp'  /opt/vmware/vcf /var/log/vmware/vcf/sddc-support/backup-2022-10-17-08-14-36-1756/vcf-backu
p-hrk01sdm001-cisdhc01-next-2022-10-17-08-14-36/configuration//opt/vmware did not complete even after waiting for 120 seconds

Cause

This issue is caused by timeouts in copying either very large files and/or copying multiple smaller files.These files normally exist because the user stored files (normally log bundles)  in locations that should not contain anything other than system files.In the log example above we can see that the SDDC Manager was in the process of copying various files just before the operation failed:
Note the last file being copied has an unusual name and is at an unusual location (SoSLogs):

2022-10-17T08:14:39.489Z [INFO commandutils.py::execute_cmd_locally::226::backup-39eaeb5e-bd78-459e-988f-098c6ab9f91c] b'vcf/SoSLogs/sos-2022-10-06-10-31-36-
3699.tar.gz\n'

Resolution

1. Running  a file-size command like du -Sh | sort -rh | head -n 15 can show the location of the largest files on the system.
2. Remove the file or files from the location mentioned in the logging and retry the backup.