vCenter backup fails after upgrading to VCSA 8.0.x

Products

• VMware vCenter Server

Issue/Introduction

Backup fails after upgrading vCenter to 8.0.1.00200 build 21860503

The error shown is:  "ERROR: stderr:Failed configuration files backup/usr/bin/tar: storage/db/vmware-observability/stellar.db: file changed as we read it".

In the /var/log/vmware/applmgmt/backup.log you see the following Errors:

[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [VCDBBackup:PID-17023] [telemetry::get_vc_version:telemetry.py:307] INFO: vCenter version is 8.0U1b
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [VCDBBackup:PID-17023] [VCDB::BackupVCDB:VCDB.py:2088] INFO: Telemetry command is ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '--data1', 'Partial Backup', '--data3', '8.0U1b', '--data2', 'Success']
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [VCDBBackup:PID-17023] [Proc::RunCmdForOutput:Proc.py:537] INFO: Executing command: ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '--data1', 'Partial Backup', '--data3', '8.0U1b', '--data2', 'Success'].
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [VCDBBackup:PID-17023] [VCDB::BackupVCDB:VCDB.py:2090] INFO: Success to push telemetry data for backup
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [VCDBBackup:PID-17023] [VCDB::BackupVCDBCleanup:VCDB.py:2118] INFO: Successfully cleaned up for VCDB backup.
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [ConfigFilesBackup:PID-17021] [Proc::GetProcsStatus:Proc.py:327] ERROR: rc: 1, stderr: /usr/bin/tar: storage/db/vmware-observability/stellar.db: file changed as we read it

[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [ConfigFilesBackup:PID-17021] [Proc::GetProcsStatus:Proc.py:345] ERROR: Process returncode is 1, but expected exit codes are [0].
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [ConfigFilesBackup:PID-17021] [ConfigFiles::BackupConfigFiles:ConfigFiles.py:336] ERROR: Failed configuration files backup- rc: 1; stdout: ; stderr:/usr/bin/tar: storage/db/vmware-observability/stellar.db: file changed as we read it
; exception:
[YYYY-MM-DDTHH:MM:SS7 [20230704-235904-21860503] [ConfigFilesBackup:PID-17021] [ConfigFiles::BackupConfigFiles:ConfigFiles.py:347] ERROR: Failed configuration files backup
Underlying process status. rc: 1
stdout:
stderr: /usr/bin/tar: storage/db/vmware-observability/stellar.db: file changed as we read it
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ConfigFiles.py", line 340, in BackupConfigFiles
    status)
util.Common.BackupRestoreError: Failed configuration files backup
Underlying process status. rc: 1
stdout:
stderr: /usr/bin/tar: storage/db/vmware-observability/stellar.db: file changed as we read it
...

[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [MainProcess:PID-16974] [BackupManager::main:BackupManager.py:591] ERROR: BackupManager encountered an exception: Hit exception inside process ConfigFilesBackup:
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [MainProcess:PID-16974] [BackupManager::HandleBackupCleanup:BackupManager.py:469] INFO: Cleaning up the backup job.
[YYYY-MM-DDTHH:MM:SS] [MainProcess:PID-16974] [BackupManager::Cleanup:BackupManager.py:405] INFO: Cleaning up all running backup child processes.
[YYYY-MM-DDTHH:MM:SS] [20230704-235904-21860503] [MainProcess:PID-16974] 

Environment

VMware vCenter Server 8.0
VMware vCenter Server 8.0.1

Cause

This issue seems to be a regression in the 8.0.U1

Resolution

This is resolved in vCenter Server 8.0 Update 1c (build number 22088981).