"Error: BackupManager encountered an exception. See logs for details." during VAMI based backup of vCenter Server with NFS protocol

Products

• VMware vCenter Server

Issue/Introduction

• VAMI scheduled backup with NFS protocol fails with error: "Error: BackupManager encountered an exception. See logs for details"

• /var/log/vmware/applmgmt/backup.log on vCenter Server

YYYY-MM-DDThh:mm:ss:uuu [VCDBBackup:PID-#####] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr: Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
    main(sys.argv[1], sys.argv[2], sys.argv[3])
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
    stdout_obj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
YYYY-MM-DDThh:mm:ss:uuu [VCDBBackup:PID-#####] [VCDB::BackupVCDB:VCDB.py:2053] ERROR: Failed to backup VCDB.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1997, in BackupVCDB
    full_db_checksum = _full_database_backup(
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1257, in _full_database_backup
    raise BackupRestoreError('Failed to dispatch full database files.',
util.Common.BackupRestoreError: Failed to dispatch full database files.
Underlying process status. rc: 28
stdout:
stderr:

 

Environment

VCSA 7.x

VCSA 8.x

Cause

The backup process failed due to insufficient space on the NFS backup storage location. The error message "ERROR: Unable to perform backup: [Errno 32] Broken pipe" indicates that the backup process was interrupted while attempting to write data to the full NFS storage, leading to a broken connection.

Resolution

vCenter Server Appliance (VCSA) file-based backup tasks are failing due to storage capacity exhaustion (out-of-space condition) on the target Network File System (NFS) repository.

Perform a comprehensive storage capacity assessment of the destination NFS export and execute remediation procedures to reclaim or provision adequate storage, ensuring the successful resumption of VCSA automated backups.

Additional Information

Proceed with the below steps to validate the available space on the NFS server

1. Log in to vCenter Server Appliance (VCSA) via ssh using root user
2. Create a temporary mount point
   
   mkdir /tmp/nfs-check
   
   
3. Mount the NFS share
   
   mount -t nfs <NFS_SERVER_IP>:/<EXPORT_PATH> /tmp/nfs-check
   
   
4. Validate available space
   
   df -h /tmp/nfs-check
   
   
5. Unmount and cleanup the remote directory
   
   umount /tmp/nfs-check
rmdir /tmp/nfs-check