vCenter Servers Security Token Service (STS) refresh in version 8.0 can cause vSphere Lifecycle Manager (vLCM) upgrade failure

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

For a long running vLCM upgrade task you may see Task abruptly cancelled with error:

com.vmware.vcIntegrity.lifecycle.TaskError.CanceledDueToServiceRestart "Task was canceled due to service restart"

Environment

VMware vCenter Server 8.0.0

Cause

In vSphere 8.0, vCenter Single Sign-On automatically renews a VMCA-generated STS signing certificate. 

The auto-renewal occurs before the STS signing certificate expires and before triggering the 90-day expiration alarm. However, in long-running upgrade or remediation tasks by using a vSphere Lifecycle Manager image on multiple ESXi hosts in a cluster, vSphere Lifecycle Manager might create a cache of STS certificates internally.

In very rare cases, if a STS certificates refresh task starts in parallel with the long-running upgrade or remediation task, the upgrade task might fail as the STS certificates in the internal cache might be different from the refreshed certificates. After the upgrade task fails, some ESXi hosts might remain in maintenance mode.

Resolution

Currently there is no resolution.

Workaround:

Manually exit any ESXi Hosts in maintenance mode and retry the upgrade or remediation. Refreshing or importing and replacing the STS signing certificates happens automatically and does not require a vCenter Server restart, to avoid downtime.