Scheduled backups fail at the 'Backup SDDC Manager Configuration' stage
Manual backups fail at same stage
sos logs contain errors similar to below:
2022-10-17T08:14:39.481Z [INFO commandutils.py::execute_cmd_locally::226::backup-39eaeb5e-####-####-####-########91c] b'vcf/OSL/open_source_license.txt\n' 2022-10-17T08:14:39.486Z [INFO commandutils.py::execute_cmd_locally::226::backup-39eaeb5e-####-####-####-########91c] b'vcf/SoSLogs/\n' 2022-10-17T08:14:39.489Z [INFO commandutils.py::execute_cmd_locally::226::backup-39eaeb5e-####-####-####-########91c] b'vcf/SoSLogs/sos-2022-10-06-10-31-36- 3699.tar.gz\n' 2022-10-17T08:16:39.660Z [ERROR commandutils.py::execute_cmd_locally::244::backup-39eaeb5e-####-####-####-########91c] Std Error : ; Command rsync -av --ex clude '*.jar' --exclude 'sddc-manager-ui-app' --exclude 'lcm/tmp' /opt/vmware/vcf /var/log/vmware/vcf/sddc-support/backup-2022-10-17-08-14-36-1756/vcf-backu p-hrk01sdm001-cisdhc01-next-2022-10-17-08-14-36/configuration//opt/vmware did not complete even after waiting for 120 seconds
Cause
This issue is caused by timeouts in copying either very large files and/or copying multiple smaller files. These files normally exist because the user stored files (normally log bundles) in locations that should not contain anything other than system files. In the log example above, we can see that the SDDC Manager was in the process of copying various files just before the operation failed: Note the last file being copied has an unusual name and is at an unusual location (SoSLogs):