Scheduled file-based backup of vCenter fails to complete due to NFS timeout

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• A customer experiences regular failures of scheduled VAMI-based vCenter backups.
• In the /var/log/vmware/applmgmt/backup.log you see the following errors:

YYYY-MM-DDT##:##:##.### [20230929-145907-21860503] [MainProcess:PID-<PID>] [BackupManager::main:BackupManager.py:570] INFO: Starting backup job...
YYYY-MM-DDT##:##:##.### [20230929-145907-21860503] [MainProcess:PID-<PID>] [BackupManager::main:BackupManager.py:577] INFO: Scheduled Backup triggerd
.
.
YYYY-MM-DDT##:##:##.### [20240427-235926-22617221] [MainProcess:PID-<PID>] [MountedFSStorageIOLib::_run_timed_cmd:MountedFSStorageIOLib.py:130] ERROR: Cmd ['/usr/bin/mount', '-t', 'nfs', '<backup server FQDN>:<vCenter backup folder>', '/storage/remote/backup/nfs/<backup server FQDN>/tqLTmul6/ANiWdShF', '-o', 'hard,intr'] failed. rc: 32, stdout: , stderr: mount.nfs: mounting <backup server FQDN>:<vCenter backup folder> failed, reason given by server: No such file or directory
.
.
YYYY-MM-DDT##:##:##.### [20230929-145907-21860503] [ComponentScriptsBackup:PID-<PID>] [Proc::GetProcsStatus:Proc.py:327] ERROR: rc: 1, stderr: Traceback (most recent call last):
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
main(sys.argv[1], sys.argv[2], sys.argv[3])
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
stdout_obj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

• Manual backups complete without error.

Environment

vCenter Server 8.0

Cause

The scheduled file-based backup coincides with other activities that take up network bandwidth from the vCenter Server Appliance. The network bottleneck on the appliance causes the connection to the NFS server to timeout.

Resolution

Adjust the schedule of the vCenter backup to not coincide with the bottleneck.