File-based backup failing with var/lib/vmware/vmca/certs.db: file changed during backup

Products

• VMware vCenter Server

Issue/Introduction

• File-based backups for vCenter Server Appliance are failing 
• When reviewing /var/log/vmware/applmgmt/backup.log, the following errors can be seen:

YYYY-MM-DDTHH:MM:SS.355 [######-######-########] [ConfigFilesBackup:PID-###] [ConfigFiles::BackupConfigFiles:ConfigFiles.py:347] ERROR: Failed configuration files backup
Underlying process status. rc: 1
stdout:
stderr: /usr/bin/tar: var/lib/vmware/vmca/certs.db: file changed as we read it
/usr/bin/tar: var/lib/vmware/vmca: file changed as we read it
Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ConfigFiles.py", line 339, in BackupConfigFiles
        raise BackupRestoreError("Failed configuration files backup",
util.Common.BackupRestoreError: Failed configuration files backup
Underlying process status. rc: 1
stdout:
stderr: /usr/bin/tar: var/lib/vmware/vmca/certs.db: file changed as we read it
/usr/bin/tar: var/lib/vmware/vmca: file changed as we read it

Environment

• VMware vCenter Server 7.0.x
• VMware vCenter Server 8.0.x
• VMware vCenter Server 9.x 

 

Cause

This issue occurs when var/lib/vmware/vmca/certs.db is updated while being compressed by a tar command as part of the file-based backup workflow.

The file var/lib/vmware/vmca/certs.db is being updated when:

• a new virtual machine (VM) with a virtual Trusted Platform Module (vTPM) device is created.
• a vTPM device is added to an existing virtual machine.
• a virtual machine that has a vTPM device configured is cloned.

When the file is updated while tar is currently accessing it, it's validity can no longer being confirmed, hence the backup workflow is getting disrupted to prevent it from creating a non-restorable backup.

Such conflicts are most like to occur in virtual desktop infrastructure (VDI) environments, where virtual machines (such as Windows 11 clients, which require vTPM) are frequently cloned.

Resolution

This issue has been partially fixed in vCenter Server 8.0 P07.
Broadcom engineering is aware of this issue and considering a permanent fix.

Note:
If any configuration files are repeatedly modified or take a long time to write, this issue may occur.
Therefore, this issue may recur in environments such as VDI even after updating vCenter Server 8.0 P07.

 

Workaround

• Reschedule backups: Ensure file-based backups run outside of windows where VMs are being created or cloned with vTPM.

• Restrict configuration: Do not attach vTPM devices to any VMs while a backup is in progress.

Additional Information

For more information regarding vTPM, please refer to What Is a Virtual Trusted Platform Module.