SDDC Manager UI fails with IDENTITY_INTERNAL_SERVER_ERROR due to Unable to obtain Security Token Service from SSO

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Attempting to login to SDDC Manager UI fails with error similar to below:
  

  Identity Internal Server Error
  IDENTITY_INTERNAL_SERVER_ERROR
  HTTP 500

• In /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log on the SDDC Manager, entries similar to the following are observed:
   
  YYYY-MM-DDThh:mm:ss -ERROR SDDC-MANAGER-UI-APP ####- [services/wrappers/requestPromiseWrapper.js, http-get-login, logAxiosError:##] axios.error.response.status {"status":500}
...
YYYY-MM-DDThh:mm:ss -ERROR SDDC-MANAGER-UI-APP ####- [services/wrappers/requestPromiseWrapper.js, http-get-login, logAxiosError:##] axios.error.response.data {"data":{"arguments":["Identity Internal Server Error"],"causes":[{"message":"Failed to get the IDP type due to Identity Internal Server Error.","type":"com.vmware.evo.sddc.common.core.error.FederatedBrokerIdentityException"},{"message":"Identity Internal Server Error","type":"com.vmware.evo.sddc.identity.model.error.PscException"},{"message":"Unable to obtain Security Token Service from SSO '<MgmtVcenterFQDN>'","type":"com.vmware.evo.sddc.common.services.psc.exception.PscException"},{"message":"Expected one or more trusted certificates, but got null","type":"java.lang.IllegalArgumentException"}],"errorCode":"SDDCMANAGER_GET_OIDC_INFO_FAILED","referenceToken":"<TOKEN>"}}
...
YYYY-MM-DDThh:mm:ss -ERROR SDDC-MANAGER-UI-APP ####- [services/wrappers/requestPromiseWrapper.js, http-get-login, logAxiosError:##] axios.error.stack {"stack":"AxiosError: Request failed with status code 500 }
  
  
• In /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log on the SDDC Manager, entries indicate the SDDC Manager is not getting the SSO token from the Management vCenter:
  
  YYYY-MM-DDThh:mm:ss ERROR [common,########,####] [c.v.e.s.i.r.a.c.v.IdentityProviderController,http-nio-127.0.0.1-7100-exec-7] Unable to get sddc manager oidc information due to Identity Internal Server Error
YYYY-MM-DDThh:mm:ss ERROR [common,########,#####] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-7] [<TOKEN>] SDDCMANAGER_GET_OIDC_INFO_FAILED 
com.vmware.evo.sddc.identity.rest.api.error.IdentityRestServiceException: 
        at com.vmware.evo.sddc.identity.rest.api.controller.v1.IdentityProviderController.getSddcWs1bOidcInfo(IdentityProviderController.java:410)
Caused by: com.vmware.evo.sddc.common.services.psc.exception.PscException: Unable to obtain Security Token Service from SSO '<MgmtVcenterFQDN>'
        at com.vmware.evo.sddc.common.util.SSOEntityService.getSamlToken(SSOEntityService.java:320)
...
Caused by: java.lang.IllegalArgumentException: Expected one or more trusted certificates, but got null
...
YYYY-MM-DDThh:mm:ss ERROR [common,#########,####] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-7100-exec-3] Unable to fetch user & groups from saml token
Caused by java.lang.RuntimeException: The SAML token signature validation failed! at com.vmware.evo.sddc.identity.sso.services.PscServiceImpl.validateSamlToken(PscServiceImpl.java:481)
...
YYYY-MM-DDThh:mm:ss.631+0000 INFO  [common,685de16daeefdb1509d7f3930c9b5c55,1a2b] [c.v.v.s.c.i.X509TrustChainKeySelector,http-nio-127.0.0.1-7100-exec-1] Failed to find trusted path to signing certificate <CN=ssoserverSign>
Caused by java.security.cert.CertPathBuilderException: Unable to find certificate chain.

Environment

• VCF 5.x
• VCF 9.x

Cause

This error is encountered when the SDDC Manager cannot validate the SAML token signature because it does not trust the Security Token Service (STS) signing certificate from the Management vCenter, or the STS certificate chain on the vCenter SSO is stale or expired.

Resolution

1. Verify the status and validity of the STS certificate on the Management vCenter.

2. If the STS certificate is valid and not expired, reboot the Management vCenter to clear any stale authentication states.

3. If the STS certificate is VMCA-signed and untrusted, add the custom CA certificates to the SDDC Manager and Common Services trust stores.

4. If the STS certificate is expired or contains a stale chain, replace it using the vCert expired certificate replacement script.

Additional Information

• How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores

• vCert - expired certificate replacement script