VAMI backups fail with "Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state."

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• When initiating a file-based backup, the backup fails with:

Db health is UNHEALTHY, Backup Failed.Disable health check to take backup in current state.

• In the log file /var/log/vmware/backup.log you can see following:

2021-10-09T23:59:49.88 [20211009-215908-18700403] [MainProcess:PID-26005] [BackupManager::main:BackupManager.py:659] ERROR: BackupManager encountered an exception: Db health is UNHEALTHY, Backup Failed.Disable health check to take backup in current state.
2021-10-09T23:59:49.88 [20211009-215908-18700403] [MainProcess:PID-26005] [BackupManager::HandleBackupCleanup:BackupManager.py:466] INFO: Cleaning up the backup job.
2021-10-09T23:59:49.89 [20211009-215908-18700403] [MainProcess:PID-26005] [BackupManager::Cleanup:BackupManager.py:402] INFO: Cleaning up all running backup child processes.​​​​​

• This error re-appears on the next day after applying the workaround.

Environment

VMware vCenter Server 7.0.3

Cause

As database schema changes are not reported to the "Database Consistency Checker" (dbcc), the tool reports the current database being in an inconsistent state. This causes the VAMI backup to fail.

This can occur post-patching vCenter 7.0 Update 3, and periodically when a vSAN service on vCenter saves historical health information in new database tables. In this case it is only a false-positive and the health of the database is not at risk.

Running the command mentioned below will refresh the latest database schema and the validation would succeed.

Resolution

This issue is resolved in vCenter Server 7.0 Update 3c.

Workaround:

Note: If the issue re-appears the next day, please see "Related information" below.

Method 1

1. Open an SSH session to the vCenter
2. Log in as root
3. Enter Shell
4. Run this command:

/usr/bin/dbcc -fbss embedded
 

5. Restart the VAMI backup, which should now complete.

Method 2

• Disable the database health check for file-based backup until the fix has been released

Additional Information

If the DB health check is enabled, the issue can re-appear the next day after applying the workaround. This is caused due to historical data saved from the vSAN service in new database tables and old getting removed, which change the database schema while the schema verification tool is not aware of.

In this case please temporarily disable the DB health check in the VAMI until a fix in a future vCenter release is available.