vCenter Server 9.0.x scheduled file-based backups fail when using 'Retain Last X Backups' policy

Products

• VMware vCenter Server
• VMware Cloud Foundation

Issue/Introduction

• After upgrading vCenter to 9.0.1 or 9.0.2, scheduled file-based backups fail consistently under specific retention configurations:

  • Manual Backups: Succeed regardless of the retention settings.

  • "Retain All Backups": Scheduled backups succeed if this option is selected.

  • "Retain Last X Backups": Scheduled backups fail consistently  when configured to retain a specific number of backups.

• Reviewing /var/log/vmware/applmgmt/backup.log in vCenter reveals failures related to:

  • Metadata dispatching and Write-Ahead Logging (WAL) file processing:

ERROR: BackupManager encountered an exception: Failed to dispatch metadata information.
ERROR: rc: 2, stderr: /usr/bin/tar: -: Wrote only 4096 of 10240 bytes
BrokenPipeError: [Errno 32] Broken pipe
util.Common.BackupRestoreError: Failed to dispatch WAL directory structure.

• • For the image builder component:

YYYY-MM-DDTHH:MM:SS [20###203-19##06-25###86] [ComponentScriptsBackup:PID-4####6] [Proc::UpdateExceptionStatus:Proc.py:408] ERROR: Checksum not generated at /dev/shm/backupRestoreSumFile-####-####-snfwa0y_
YYYY-MM-DDTHH:MM:SS [20###203-19##06-25###86] [ComponentScriptsBackup:PID-4####6] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:127] ERROR: Component imagebuilder backup dispatch failed.
YYYY-MM-DDTHH:MM:SS [20###203-19##06-25###86] [ComponentScriptsBackup:PID-4####6] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:148] ERROR: Failed to dispatch component imagebuilder backup
Underlying process status. rc: 255
stdout:
stderr:
Traceback (most recent call last):   File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ComponentScripts.py", line 128, in ComponentScriptsBackup raise BackupRestoreError(('Failed to dispatch component %s 'util.Common.BackupRestoreError: Failed to dispatch component imagebuilder backup

• • For the vum Component:

ERROR: Checksum not generated at /dev/shm/backupRestoreSumFile-
ERROR: Unable to perform backup: [Errno 32] Broken pipeException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
ERROR: BrokenPipeError: [Errno 32] Broken pipe
ERROR: Component backup command "/etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup" failed 120.
ERROR: Error during component vum backup

Environment

• VMware Cloud Foundation (VCF) 9.0.x
• VMware vCenter Server 9.0.x

Cause

• This is a known issue involving the contents of the backup retention file and how metadata is dispatched during the retention cleanup phase.
• Configuring scheduled backups with a specific retention count "Retain last X Backups" is currently triggering backup failures.

Resolution

This issue is fixed in VCF 9.1 version and higher. 

See  VMware Cloud Foundation 9.1. for steps to download this release.

Workaround

If an upgrade to version 9.1 is not immediately possible, implement the following steps.

Note: These steps are applicable only to VCF 9.0.x. Before proceeding with the changes below, take a non-memory snapshot of the vCenter VM

1. Download and Upload the attached files to the /root directory on the vCenter:

   ScheduleManager.py
   SchedulerCron.py

2. SSH to the vCenter Server as root.
3. Make a copy of the existing backup script files:

   cp /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/ScheduleManager.py /root/ScheduleManager.py.original
   cp /usr/lib/applmgmt/backup_restore/scripts/SchedulerCron.py /root/SchedulerCron.py.original

4. Replace these files with the two scripts attached to this article:

   cp /root/ScheduleManager.py /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/
   cp /root/SchedulerCron.py /usr/lib/applmgmt/backup_restore/scripts/

5. Set Permissions 

   chmod 544 /usr/lib/applmgmt/backup_restore/scripts/SchedulerCron.py
   chmod 444 /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/ScheduleManager.py

6. Restart services
   

   service-control --restart applmgmt

After performing the above steps, the scheduled backups and retention should work as expected.

Additional Information

Scheduled backups of vCenter Server fail with error "Failed to dispatch metadata information"

vCenter vami backups to storage cluster fail with error "deprecated key derivation used"