Troubleshooting vCenter Server Appliance (VCSA) File-Based Backup Failures and VCDB Backup Errors (VCSA)

Products

• VMware vCenter Server

Issue/Introduction

vCenter Server Appliance file-based backups (SFTP/SCP/NFS) fail under two primary scenarios, typically following an emergency evacuation, cloning, or migration activity:

1. Snapshot Lock Failure
   • Manual and scheduled backups fail with the error:
     BackupManager encountered an exception. Exception: Full backup not allowed during VM snapshot.

   • log entries in vCenter /var/log/vmware/applmgmt/backup.log

     [YYYY-MM-DDTHH:MM:SS].648 [VCDBBackup:PID-####] [VCDB::BackupVCDB:VCDB.py:2060] ERROR: Encounter error during backup VCDB. Traceback (most recent call last): File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1939, in BackupVCDB raise Exception('Full backup not allowed during VM snapshot') Exception: Full backup not allowed during VM snapshot
     [YYYY-MM-DDTHH:MM:SS].746 [MainProcess:PID-####] [BackupManager::Cleanup:BackupManager.py:406] ERROR: Failed to clean up backup child processes. Exception: Hit exception inside process VCDBBackup: Full backup not allowed during VM snapshot

     
     This occurs even when no snapshots are present in the vSphere inventory.
2. Scheduler Validation Failure
   • Scheduled backups fail immediately, while manual backups to the same destination succeed. Logs in /var/log/vmware/applmgmt/backup.log may indicate generic validation failures or metadata mismatches.

Environment

VMware vCenter Server Appliance 7.x  
VMware vCenter Server Appliance 8.x  
VMware vCenter Server Appliance 9.x  

Cause

The backup engine utilizes local and remote sentinel files to ensure data consistency and prevent concurrent operations. These files can become stale during non-standard shutdowns or cloning operations:

• Local Stale Marker
  • A zero-byte file named backupMarker.txt located in /etc/vmware/ persists post-migration, or after the recovery of the vCenter Virtual Machine after restoration from a third party back up solution (Commvault, Veeam, etc.). This file signals to the backup engine that a snapshot-based operation is already in progress, blocking new full backups File-Based backup in vCenter VAMI UI fails with "Full backup not allowed during VM snapshot"
    
    
• Remote Metadata Mismatch
  • The scheduled backup engine performs a version and builds validation against the backup-metadata.json file at the root of the backup target. If the VCSA IP has changed or the version recorded in the metadata does not match the current build, the scheduler refuses to initialize the job Scheduled vCenter File-Based Backup fails but manual backup jobs succeed

Resolution

Resolving Snapshot Marker Errors

1. Access the VCSA via SSH using root credentials.
2. Navigate to the configuration directory:
   cd /etc/vmware
   
   
3. Verify the existence of the marker:
ls -l backupMarker.txt
   
   
4. Delete the stale file:
   rm /etc/vmware/backupMarker.txt
   
   
5. Retry the backup from the VAMI (Port 5480)

Resolving Scheduled Backup Metadata Mismatch

1. Log in to the backup storage server (SFTP/SCP target) via SSH or console.
2. Navigate to the root directory where the VCSA backups are stored.
3. Locate the file backup-metadata.json
4. Rename the file to allow the scheduler to create a fresh index:
   mv backup-metadata.json backup-metadata.json.old
   
   
5. In the VAMI Backup interface, edit and save the schedule to trigger a validation check. Perform a manual backup to confirm connectivity, then verify the next scheduled run.

Additional Information

For further details on identifying active VM snapshots that may be locked at the ESXi host level, refer to Finding and listing virtual machine snapshots