VAMI backup with SMB protocol fails at 55% for vCenter Server 8.0 U3

Products

• VMware vCenter Server

Issue/Introduction

• VAMI backup via SMB fails at approximately 55%.

In vCenter at /var/log/vmware/applmgmt/backup.log, the following events can be observed. 

YYYY-MM-DDTHH:MM:SS.413 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr:
/usr/bin/tar: var/lib/rbd/cache/vital/6f83ab066e919694854230b5b#######: File removed before we read it
...
/usr/bin/tar: var/lib/rbd/cache/vital: file changed as we read it
YYYY-MM-DDTHH:MM:SS.414 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is 1, but expected exit codes are [0].
YYYY-MM-DDTHH:MM:SS.414 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [Proc::GetProcsStatus:Proc.py:361] ERROR: rc: 0, stderr: b'putting file - as \\<vCenter_backup_location>\\config_files.tar.gz (25370.3kb/s) (average 25370.3 kb/s)\n'
YYYY-MM-DDTHH:MM:SS.414 [########-152459-24322831] [ConfigFilesBackup:PID-#######] [ConfigFiles::BackupConfigFiles:ConfigFiles.py:332] ERROR: Failed configuration files backup- rc: 1; stdout: ; stderr:/usr/bin/tar:
/var/lib/rbd/cache/vital/6f83ab066e919694854230b5b#######: File removed before we read it
/usr/bin/tar: var/lib/rbd/cache/vital/ff6cc069ec3cc0a18503e32528######: File removed before we read it
...
/usr/bin/tar: var/lib/rbd/cache/vital: file changed as we read it; exception:

Environment

VMware vCenter Server 8.x

Cause

This issue occurs as VAMI is unable to write to /var/lib/rbd/cache/vital/##### files.

Resolution

Make the following changes to address the issue:

1. Access the vCenter appliance shell as root user
   
   
2. Backup the RBD manifest: 
   
cp /etc/vmware/backup/manifests/rbd.json /etc/vmware/backup/manifests/rbd.json_bak
   
   
3. Remove the RBD manifest file: 
   
   rm /etc/VMware/backup/manifests/rbd.json


Workaround:

Note: This workaround excludes Auto Deploy data from the backup.

1. Access the vCenter appliance shell root user
   
   
2. Stop the RBD watchdog service before initiating a manual backup: 
   
   systemctl stop vmware-rbd-watchdog