Authentication stopped working on vCenter Server and rebooting the appliance resolves the issue

Products

• VMware vCenter Server

Issue/Introduction

• Authentication stops working on vCenter Server and users are unable to login to vSphere Client.
• VPXD service log shows log entries as below:
  
  /var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS.###Z error vpxd[<pid>] [Originator@6876 sub=SsoWrapper] [AcquireToken] AcquireToken exception: N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:InvalidTimeRange; request failed.)
YYYY-MM-DDTHH:MM:SS.###Z error vpxd[<pid>] [Originator@6876 sub=sms] [ConnectLocked] Failed to login to service: N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:InvalidTimeRange; request failed.)
  
  
• Authentication starts working after rebooting the vCenter Server.

Environment

• vCenter Server 8.x.
• vCenter 9.x.

Cause

• This issue is caused due to STS (Secure Token Signing) certificate expiry on vCenter. 
• vCenter has a feature to automatically renew the STS certificate before the expiry. However, this will be triggered only if the STS certificate is issued by the active VMCA Root on the vCenter. If the STS is issued by an old PSC or any other CA, the auto renew of the STS certificate will not work.
• SSO log shows entries as below:
  
  /var/log/vmware/sso/vmware-identity-sts.log
  
  YYYY-MM-DDTHH:MM:SS.###Z INFO sts[90:tomcat-http--44] [CorId=<id>] [com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Day Month DD HH:MM:SS GMT YYYY, endTime=Day Month DD HH:MM:SS GMT YYYY] :: Signing certificate is not valid at Day Month DD HH:MM:SS GMT YYYY, cert validity: TimePeriod [startTime=Day Month DD HH:MM:SS GMT YYYY, endTime=Day Month DD HH:MM:SS GMT YYYY]
  
  
• Trust Management Service log shows that Signing certificate will not automatically be renewed before expiration as the certificate was not issued by the same vCenter Server.
  
  /var/log/vmware/trustmanagement/trustmanagement-svcs.log

YYYY-MM-DDTHH:MM:SS.###Z [signingCertAutoRenewScheduler-1 [] INFO  com.vmware.vcenter.trustmanagement.signingcertmgmt.SigningCertificateAutoRenew  opId=] subject "C=US,CN=ssoserverSign\,dc\=vsphere\,dc\=local", issuer "C=US,CN=CA\, CN\=<Old PSC Name>\, dc\=vsphere\,dc\=local", notBefore Day Month DD HH:MM:SS GMT YYYY, notAfter Day Month DD HH:MM:SS GMT YYYY
YYYY-MM-DDTHH:MM:SS.###Z [signingCertAutoRenewScheduler-1 [] INFO  com.vmware.vcenter.trustmanagement.signingcertmgmt.SigningCertificateAutoRenew  opId=] Signing certificate chain expires on Day Month DD HH:MM:SS GMT YYYY
YYYY-MM-DDTHH:MM:SS.###Z [signingCertAutoRenewScheduler-1 [] WARN  com.vmware.vcenter.trustmanagement.signingcertmgmt.SigningCertificateAutoRenew  opId=] Signing certificates will *not* automatically be renewed before expiration.

Resolution

Reboot the vCenter Server to trigger the STS certificate expiry check and forcefully initiates automatic renewal of the STS certificate.

Additional Information

• STS certificate expiry check is triggered every 24 hours. In case reboot is not triggered, STS will get auto renewed during the next certificate check interval within 24 hours.
• Trust Management service will show log entries as below for the automatic renewal:
  
  /var/log/vmware/trustmanagement/trustmanagement-svcs.log

YYYY-MM-DDTHH:MM:SS.###Z [signingCertAutoRenewScheduler-1 [] INFO  com.vmware.vcenter.trustmanagement.signingcertmgmt.SigningCertificateAutoRenew  opId=] Signing certificate chain expires on Day Month DD HH:MM:SS GMT YYYY
YYYY-MM-DDTHH:MM:SS.###Z [signingCertAutoRenewScheduler-1 [] WARN  com.vmware.vcenter.trustmanagement.signingcertmgmt.SigningCertificateAutoRenew  opId=] Signing certificate chain is expired. Forcing renewal.
YYYY-MM-DDTHH:MM:SS.###Z [signingCertAutoRenewScheduler-1 [] WARN  com.vmware.vcenter.trustmanagement.signingcertmgmt.SigningCertificateAutoRenew  opId=] Renewing signing certificate chain.
YYYY-MM-DDTHH:MM:SS.###Z [signingCertAutoRenewScheduler-1 [] INFO  com.vmware.vcenter.trustmanagement.signingcertmgmt.SigningCertificateAutoRenew  opId=] Signing certificate chain has been renewed.