Scheduled vCenter File-Based Backup fails but manual backup jobs succeed

Products

• VMware vCenter Server

Issue/Introduction

• Scheduled file-based backups configured in the VAMI (https://vcenter-fqdn:5480) consistently fail and may display a generic error message.
  • Manual backups (Backup Now) to the exact same destination and using the same credentials complete successfully.
    
    
• The VAMI activity interface may display generic errors or informational disclaimers about the cause of the failed backup jobs.
  
  
• The backup destination is a Linux-based server
  • (e.g., Red Hat, Ubuntu) using SFTP, SCP, or NFS.
    
    
• Failures may start right after upgrading the vCenter Server.

Cause

• This issue is caused by stale or incompatible metadata residing on the backup server potentially due failed backup job
  • This often correlates to failed backups that occurred on a pre-upgrade version of vCenter.
    
    
• When a Manual Backup is performed, vCenter treats the operation as a standalone event.
  • However, the Scheduled Backup engine performs a validation check against the existing directory structure and the backup-metadata.json file located at the root of the backup target.
    
    
• If the vCenter version recorded in the existing backup-metadata.json does not match the current post-upgrade build, the scheduler may fail to initialize the job or refuse to overwrite/append to the existing backup chain, perceiving a version mismatch.
  • If an upgrade was not performed, invalid data on the backup-metadata.json file or being unable to read or write to the file properly can also result in the same issues.

Resolution

To resolve this issue, the existing metadata file must be cleared to allow the upgraded vCenter Scheduler to initialize a fresh backup index.

1. Log in to the Backup Server via SSH or console.
   
   
2. Navigate to the root directory where the vCenter backups are stored.
   
   
3. Locate the file named backup-metadata.json.
   
   
4. Rename the file to effectively "hide" it from the vCenter scheduler:
   mv backup-metadata.json backup-metadata.json.old


5. In the VAMI Backup page, use the Edit button and set the start time for a scheduled backup job to begin right away (a minute or two after the current time), or wait for the next scheduled time to verify the scheduled jobs now complete successfully.

Additional Information

• (KB 414580) VCSA after upgrade to version 8 is not able to complete backup describes a similar failure which also impacts manual backup jobs.
  • The issue described in that article is due to losing connection to the backup server and applies if the vCenter's fileTransferMode is set to "parallel".
    
    
  • If scheduled backup jobs continue to fail or if connectivity issues are suspected, refer to the information documented there.