'BackupManager encountered an exception. See logs for details' error while backing VC VAMI backup with NFS protocol.

Products

• VMware vCenter Server

Issue/Introduction

• VAMI scheduled backup with NFS protocol fails with error: "BackupManager encountered an exception."

• Example log entries from vCenter Server, similar to the following. vCenter log location: /var/log/vmware/applmgmt/backup.log

 

YYYY-MM-DDThh:mm:ss:uuu [VCDBBackup:PID-214660] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr: Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
    main(sys.argv[1], sys.argv[2], sys.argv[3])
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
    stdout_obj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

YYYY-MM-DDThh:mm:ss:uuu [VCDBBackup:PID-214660] [VCDB::BackupVCDB:VCDB.py:2053] ERROR: Failed to backup VCDB.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1997, in BackupVCDB
    full_db_checksum = _full_database_backup(
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1257, in _full_database_backup
    raise BackupRestoreError('Failed to dispatch full database files.',
util.Common.BackupRestoreError: Failed to dispatch full database files.
Underlying process status. rc: 28
stdout:
stderr:

 

Environment

vCenter Server 7.x

vCenter Server 8.x

VMware Cloud Foundation 5.x

VMware Cloud Foundation 4.x

Cause

The backup process failed due to insufficient space on the NFS backup storage location. The error message "ERROR: Unable to perform backup: [Errno 32] Broken pipe" indicates that the backup process was interrupted while attempting to write data to the full NFS storage, leading to a broken connection.

Resolution

Disk Space Management:

1. Monitor Disk Usage: Set up monitoring to track the NFS datastore's free space. Tools like vCenter Server's Capacity Planner or third-party monitoring solutions can help.
Automate Cleanup: Implement scripts or scheduled tasks to periodically remove old backups from the NFS datastore. This can be done using the find command, or with dedicated backup tools that offer file retention policies.
Consider Disk Expansion: If your datastore is consistently nearing full capacity, consider expanding the NFS server's storage or allocating a dedicated datastore for VAMI backups to prevent contention with other workloads.

2. Backup Strategy:

Incremental Backups: Use incremental backups to only backup changed data. This can significantly reduce the size of backup files, saving disk space on the NFS datastore.
Compressed Backups: Compress backup files before storing them. This can reduce the size of backups even further, requiring less storage space.
Backup Rotation Policies: Define a backup rotation policy (e.g., keep the last 3 daily backups, weekly backups for the past 4 weeks, and monthly backups for the past year). This ensures that older backups are removed and only necessary data is retained, freeing up space.

3. Backup Tools and Features:

Backup Tools: Investigate using dedicated backup solutions like "third party" that offer features like:
Data Deduplication: Eliminate redundant data blocks within the backups, saving significant storage space.
Data Compression: Compress the backup data before storing it.
Backup Rotation: Implement robust backup rotation policies to manage data retention.
vCenter Server Backup Features: Explore the backup features within vCenter Server itself. It may offer options for incremental backups, retention policies, or integrations with third-party backup tools.

4. Additional Considerations:

Backup Location: Consider if there's a way to change the VAMI backup location to a datastore with more available space, especially if the current NFS datastore is used for other workloads.
Networking: Ensure your network connection between vCenter Server and the NFS datastore is sufficient to handle backup traffic without causing bottlenecks.

By implementing these recommendations, you can effectively manage disk space on your NFS datastore, prevent backup failures due to full disk space, and ensure reliable VAMI backups. Remember to carefully analyze your backup needs, data retention policies, and available resources to choose the best approach for your specific environment.