vCenter VAMI Scheduled Backup Failed Intermittently - Error: 'Failed to backup VCDB'
search cancel

vCenter VAMI Scheduled Backup Failed Intermittently - Error: 'Failed to backup VCDB'

book

Article ID: 392109

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vCenter VAMI scheduled Backup failing Intermittently, however, manual backups works fine.
  • VAMI backup fails with an error
    Failed to backup VCDB
  • In vCenter /var/log/vmware/applmgmt/backup.log reports the below error: 
    [VCDBBackup:PID-2######2] [backupRestoreDispatch::dispatchFiles:backupRestoreDispatch.py:291] INFO: tarCmd = ['/usr/bin/tar', '-cz', '-C', '/', '--ignore-failed-read', '--no-recursion', '--warning', 'no-file-ignored', 'storage/dblog/vpostgres/pg_xlog','storage/dblog/vpostgres/pg_xlog/archive_status']
[VCDBBackup:PID-2######2] [Proc::GetProcsStatus:Proc.py:327] ERROR: rc: 0, stderr: *** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

[VCDBBackup:PID-2######2] [VCDB::BackupVCDB:VCDB.py:2053] ERROR: Failed to backup VCDB.

File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 2004, in BackupVCDB
wal_struct_checksum = _wal_dir_structure_backup(
File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1181, in _wal_dir_structure_backup
raise BackupRestoreError('Failed to dispatch WAL directory structure.',
util.Common.BackupRestoreError: Failed to dispatch WAL directory structure.

[VCDBBackup:PID-2######2] [VCDB::BackupVCDB:VCDB.py:2070] INFO: Terminate sub process 2892054
[VCDBBackup:PID-2######2] [telemetry::check_telemetry_enabled:telemetry.py:285] INFO: Check the telemetry conf file
[VCDBBackup:PID-2######2] [telemetry::check_telemetry_enabled:telemetry.py:289] INFO: Check configuration parameter vcenter.telemetry
[VCDBBackup:PID-2######2] [telemetry::check_telemetry_enabled:telemetry.py:293] INFO: Telemetry for Postgres is enabled
[VCDBBackup:PID-2######2] [telemetry::get_vc_version:telemetry.py:301] INFO: Get vCenter version from /etc/vmware/.buildInfo
[VCDBBackup:PID-2######2] [telemetry::get_vc_version:telemetry.py:307] INFO: vCenter version is 8.0U2a
[VCDBBackup:PID-2######2] [VCDB::BackupVCDB:VCDB.py:2088] INFO: Telemetry command is ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '--data1', 'Full Backup', '--data3', '8.0U2a', '--data2', 'Fail']
[VCDBBackup:PID-2######2] [Proc::RunCmdForOutput:Proc.py:537] INFO: Executing command: ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '--data1', 'Full Backup', '--data3', '8.0U2a', '--data2', 'Fail'].
[VCDBBackup:PID-2######2] [VCDB::BackupVCDB:VCDB.py:2090] INFO: Success to push telemetry data for backup
[VCDBBackup:PID-2######2] [VCDB::BackupVCDBCleanup:VCDB.py:2109] INFO: Finishing full database backup
[VCDBBackup:PID-2######2] [VCDB::BackupVCDBCleanup:VCDB.py:2118] INFO: Successfully cleaned up for VCDB backup.
[MainProcess:PID-2891890] [Proc::VerifyProcStatusAndGetArchive:Proc.py:158] ERROR: Error at process VCDBBackup; rc:255.
[MainProcess:PID-2891890] [Proc::VerifyProcStatusAndGetArchive:Proc.py:162] ERROR: stderr:Failed to dispatch WAL directory structure.

[BackupManager::main:BackupManager.py:591] ERROR: BackupManager encountered an exception: Hit exception inside process VCDBBackup:
[BackupManager::HandleBackupCleanup:BackupManager.py:469] INFO: Cleaning up the backup job.
[BackupManager::Cleanup:BackupManager.py:405] INFO: Cleaning up all running backup child processes.
[BackupManager::Cleanup:BackupManager.py:412] INFO: Cleaning up all sizeFiles.
[BackupManager::Cleanup:BackupManager.py:420] INFO: Cleaning up ConfigFiles.
[BackupManager::Cleanup:BackupManager.py:420] INFO: Cleaning up ComponentScripts.
[BackupManager::Cleanup:BackupManager.py:420] INFO: Cleaning up VCDB.
[VCDB::BackupVCDBCleanup:VCDB.py:2109] INFO: Finishing full database backup
[VCDB::BackupVCDBCleanup:VCDB.py:2118] INFO: Successfully cleaned up for VCDB backup.
[BackupManager::Cleanup:BackupManager.py:420] INFO: Cleaning up Lotus.
[Lotus::BackupLotusCleanup:Lotus.py:140] INFO: Successfully completed Lotus cleanup.
[BackupManager::Cleanup:BackupManager.py:420] INFO: Cleaning up VTSDB.
[VTSDB::BackupVTSDBCleanup:VTSDB.py:237] INFO: Successfully cleaned up for VTSDB backup.
[BackupManager::Cleanup:BackupManager.py:420] INFO: Cleaning up StellarDb.
[StellarDb::BackupStellarDbCleanup:StellarDb.py:131] INFO: Successfully completed Stellar cleanup.
[BackupManager::Cleanup:BackupManager.py:429] INFO: Cleaning up failed backup files.
[BackupManager::HandleBackupCleanup:BackupManager.py:472] INFO: Cleaned up the backup job.
[vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
[BackupManager::main:BackupManager.py:618] INFO: Backup job failed.

 

Environment

VMware vCenter Server 

Cause

The failure of vCenter VAMI backups is frequently linked to an unresponsive telemetry service. During the backup's initial service verification phase, a telemetry service stuck in a transitional state (typically resulting from an interrupted update) or lacking topology configuration prevents the system from achieving the stability needed for data export. Consequently, the backup engine waits indefinitely for the service to reach a "Running" status, resulting in a stalled or failed backup job.

 

 

 

Resolution

To resolve the issue with vpostgres telemetry configuration needs to be edited. Contact Broadcom support for assistance - Refer Contact Broadcom support

Powered by Wolken Software