Error: "BackupManager encountered an exception" or scheduled vCenter backup failures in VCF 9

Products

• VMware vCenter Server

Issue/Introduction

• vCenter backup fails with the error: "BackupManager encountered an exception" when initiated from the VAMI.
• Scheduled backups do not start as configured.
• Log files /var/log/vmware/applmgmt/backup.log and /var/log/vmware/applmgmt/backupSchedulerCron.log contain a SoapException.

YYYY-MM--DD HH:MM:SS,888 ##### Issuing the scheduled backup request for schedule: default.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/VapiClientHelper.py", line 76, in File "/usr/lib/vmware/site-packages/pyVim/ssov2.py", line 75, in get_hok_saml_assertion_for_service_user
   raise SoapException(fault, *parsed_fault)
   pyVim.sso.SoapException: SoapException:
   faultcode: ns0:RequestFailed
   faultstring: EndTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY is not after startTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY

YYYY-MM--DD HH:MM:SS.608 [###] [ComponentScriptsBackup:PID-###] [Log::run:Log.py:64] ERROR: Failed to get HoK token with error SoapException:
YYYY-MM--DD HH:MM:SS.609 [###] [ComponentScriptsBackup:PID-###] [Log::run:Log.py:64] ERROR: faultcode: ns0:RequestFailed
YYYY-MM--DD HH:MM:SS.609 [###] [ComponentScriptsBackup:PID-###] [Log::run:Log.py:64] ERROR: faultstring: EndTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY is not after startTime: <Day> <Month> <Date> HH:MM:SS GMT YYYY

Environment

• VMware vCenter Server 9.0.x
• VMware Cloud Foundation 9.1

Cause

When the scheduled backup starts from vCenter VAMI, the BackupManager creates a security token using the local time zone. Because the Single Sign-On (SSO) service expects this time to be in UTC, the time zone difference causes the security check to fail. The system rejects the request because it incorrectly calculates the end time as happening before the start time.

Resolution

Broadcom Engineering is aware of this issue, and a fix is planned for a future release. Pease subscribe to this article to be updated on the fix status. For more information on subscribing to articles, see Subscribe to a Broadcom knowledge article by article or product

Workaround:

Set the vCenter server appliance time zone to UTC.

1. Log in to the vCenter VAMI at https://<vCenter_FQDN>:5480

2. Navigate to Time > Time Zone
3. Click Edit and change the time zone to UTC.
4. Restart the applmgmt service from VAMI: Services > Appliance Management Service > restart
5. Re-attempt the scheduled backup.

Additional Information

To speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.