VAMI backups fail with "Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state."
book
Article ID: 318143
calendar_today
Updated On:
Products
VMware vCenter Server
Issue/Introduction
Symptoms:
When initiating a file-based backup, the backup fails with:
Db health is UNHEALTHY, Backup Failed.Disable health check to take backup in current state.
In the log file /var/log/vmware/backup.log you can see following:
2021-10-09T23:59:49.88 [20211009-215908-18700403] [MainProcess:PID-26005] [BackupManager::main:BackupManager.py:659] ERROR: BackupManager encountered an exception: Db health is UNHEALTHY, Backup Failed.Disable health check to take backup in current state. 2021-10-09T23:59:49.88 [20211009-215908-18700403] [MainProcess:PID-26005] [BackupManager::HandleBackupCleanup:BackupManager.py:466] INFO: Cleaning up the backup job. 2021-10-09T23:59:49.89 [20211009-215908-18700403] [MainProcess:PID-26005] [BackupManager::Cleanup:BackupManager.py:402] INFO: Cleaning up all running backup child processes.
This error re-appears on the next day after applying the workaround.
Environment
VMware vCenter Server 7.0.3
Cause
As database schema changes are not reported to the "Database Consistency Checker" (dbcc), the tool reports the current database being in an inconsistent state. This causes the VAMI backup to fail.
This can occur post-patching vCenter 7.0 Update 3, and periodically when a vSAN service on vCenter saves historical health information in new database tables. In this case it is only a false-positive and the health of the database is not at risk.
Running the command mentioned below will refresh the latest database schema and the validation would succeed.
Resolution
This issue is resolved in vCenter Server 7.0 Update 3c.
Workaround:
Note: If the issue re-appears the next day, please see "Related information" below.
Method 1
Open an SSH session to the vCenter
Log in as root
Enter Shell
Run this command:
/usr/bin/dbcc -fbss embedded
Restart the VAMI backup, which should now complete.
Method 2
Disable the database health check for file-based backup until the fix has been released
Additional Information
If the DB health check is enabled, the issue can re-appear the next day after applying the workaround. This is caused due to historical data saved from the vSAN service in new database tables and old getting removed, which change the database schema while the schema verification tool is not aware of.
In this case please temporarily disable the DB health check in the VAMI until a fix in a future vCenter release is available.