vCenter Server scheduled backups fail with "BackupManager encountered an exception" or "device is busy" in Linked Mode environments

Products

• VMware vCenter Server

Issue/Introduction

• Scheduled file-based backups of vCenter Server Appliances in Enhanced Linked Mode fail intermittently.
• Manual backups through the VAMI is working.
• In /var/log/vmware/applmgmt/backup.log you see similar entries 

[MainProcess:PID-######] [MountedFSStorageIOLib::unmount:MountedFSStorageIOLib.py:278] ERROR: Failed to unmount the nfs share at /storage/remote/backup/nfs/NFS_FQDN/#####/###, with flags: ; Err- rc: 16, out: , err: umount.nfs4: 
/storage/remote/backup/nfs/NFS_FQDN/#####/#####: device is busy


[MainProcess:PID-######] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Plugin error occurred. ErrCode: 153, Args: ()

Environment

vCenter 8.x

Cause

This issue typically occurs due to concurrent access conflicts on the shared NFS backup target. When multiple vCenter nodes in a Linked Mode environment attempt to validate the backup location or perform unmount operations at the exact same time, a race condition or resource locking on the NFS server can cause the "device is busy" or "No such file or directory" errors.

Resolution

To resolve this issue, stagger the backup start times for each vCenter node to ensure they do not overlap during the validation and execution phases.

1. Log in to each vCenter Server Appliance Management Interface at https://<vcenter-fqdn>:5480
2. Navigate to Backup > Schedule
3. Edit the backup schedule for each vCenter
4. Introduce a minimum delay of 30 to 60 minutes between the start times of different vCenter nodes (e.g., Node 1 at 00:00, Node 2 at 01:00)
5. Ensure the NFS server has no stale lock files or sessions related to the failed backup attempts.