Error: "BackupManager encountered an exception" or scheduled backups do not start in vCenter (VCF 9.1)

Products

• VMware vCenter Server

Issue/Introduction

• When initiating a vCenter backup from VAMI on vCenter, the process fails with the error: “BackupManager encountered an exception”
• vCenter VAMI scheduled backups do not trigger.
• Following log snippets are observed in the vCenter under /var/log/vmware/applmgmt/backup.log and /var/log/vmware/applmgmt/backupSchedulerCron.log respectively:

YYYY-MM--DD HH:MM:SS,888 ##### Issuing the scheduled backup request for schedule: default.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/VapiClientHelper.py", line 76, in File "/usr/lib/vmware/site-packages/pyVim/ssov2.py", line 75, in get_hok_saml_assertion_for_service_user
   raise SoapException(fault, *parsed_fault)
   pyVim.sso.SoapException: SoapException:
   faultcode: ns0:RequestFailed
   faultstring: EndTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY is not after startTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY

YYYY-MM--DD HH:MM:SS.608 [###] [ComponentScriptsBackup:PID-###] [Log::run:Log.py:64] ERROR: Failed to get HoK token with error SoapException:
YYYY-MM--DD HH:MM:SS.609 [###] [ComponentScriptsBackup:PID-###] [Log::run:Log.py:64] ERROR: faultcode: ns0:RequestFailed
YYYY-MM--DD HH:MM:SS.609 [###] [ComponentScriptsBackup:PID-###] [Log::run:Log.py:64] ERROR: faultstring: EndTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY is not after startTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY

Environment

vCenter 9.x
VMware Cloud Foundation (VCF) 9.1

Cause

When the backup starts, the system creates a security token using the local time zone. Because the Single Sign-On (SSO) service expects this time to be in UTC, the time zone difference causes the security check to fail. The system rejects the request because it incorrectly calculates the end time as happening before the start time.

Resolution

Broadcom Engineering is aware of this issue, and a fix is planned for a future release.

Workaround:

Set the vCenter server appliance time zone to UTC.

1. Log in to the vCenter VAMI at https://<vCenter_FQDN>:5480
2. Navigate to Time > Time Zone
3. Click Edit and change the time zone to UTC.
4. Restart the applmgmt service from VAMI: Services > Appliance Management Service > restart
5. Re-attempt the scheduled backup.