VAMI backup fails with "Error: BackupManager encountered an exception. See logs for details"

Products

• VMware vCenter Server

Issue/Introduction

• VAMI/File-based backup fails at approximately 90% (SFTP) or 55% (SMB) in vCenter Server 8.0 U3.

In vCenter at /var/log/vmware/applmgmt/backup.log, the following events can be observed. 

YYYY-MM-DDTHH:MM:SS.413 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr:
/usr/bin/tar: var/lib/rbd/cache/vital/6f83ab066e919694854230b5b#######: File removed before we read it
...
/usr/bin/tar: var/lib/rbd/cache/vital: file changed as we read it
YYYY-MM-DDTHH:MM:SS.414 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is 1, but expected exit codes are [0].
YYYY-MM-DDTHH:MM:SS.414 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [Proc::GetProcsStatus:Proc.py:361] ERROR: rc: 0, stderr: b'putting file - as \\<vCenter_backup_location>\\config_files.tar.gz (25370.3kb/s) (average 25370.3 kb/s)\n'
YYYY-MM-DDTHH:MM:SS.414 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [ConfigFiles::BackupConfigFiles:ConfigFiles.py:332] ERROR: Failed configuration files backup- rc: 1; stdout: ; stderr:/usr/bin/tar:
/var/lib/rbd/cache/vital/6f83ab066e919694854230b5b#######: File removed before we read it
/usr/bin/tar: var/lib/rbd/cache/vital/ff6cc069ec3cc0a18503e32528######: File removed before we read it
...
/usr/bin/tar: var/lib/rbd/cache/vital: file changed as we read it; exception:

Environment

VMware vCenter Server 8.0 U3

Cause

This issue occurs as VAMI is unable to write to /var/lib/rbd/cache/vital/##### files.

Resolution

Make the following changes to address the issue:

1. Access the vCenter appliance shell as root user
   
   
2. Backup the RBD manifest: 
   
cp /etc/vmware/backup/manifests/rbd.json /etc/vmware/backup/manifests/rbd.json_bak
   
   
3. Remove the RBD manifest file: 
   
   rm /etc/vmware/backup/manifests/rbd.json


Workaround:

Note: These workaround excludes Auto Deploy data from the backup.

Option 1: Stopping the RBD watchdog service

1. Access the vCenter appliance shell root user
   
   
2. Stop the RBD watchdog service before initiating a manual backup: 
   
   systemctl stop vmware-rbd-watchdog

Option 2: Moving/commenting out the vital file reported in the error in RDB Manifest file

This method excludes the vital files from being attempted to be backed up during the backup job.

1. Log in to the vCenter appliance shell as root.
2. Navigate to the backup manifests directory and create a backup of the rbd.json file (cp /etc/vmware/backup/manifests/rbd.json /etc/vmware/backup/manifests/rbd.json_bak)
3. Use vi editor to comment (#) out the vital file:
   • virbd.json
   • Press 'I' or insert key to make changes.
   • Remove or comment the line containing '/var/lib/rbd/cache/vital'
   • Save the file by pressing the ESC key and typing :wq!
4. Retry the backup