Logging into SDDC Manager fails with IDENTITY_INTERNAL_SERVER

Products

VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

After authenticating, logging into SDDC Manager shows a message similar to:

{"message":"500 - \"{\\\"errorCode\\\":\\\"IDENTITY_INTERNAL_SERVER_ERROR\\\",\\\"arguments\\\":[],\\\"message\\\":\\\"Identity Internal Server Error\\\",\\\"referenceToken\\\":\\\"ABC123\\\"}\""}

On the SDDC Manager you see entries similar to this in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

VERBOSE [5b08949e26874c68] [services/sso-initialization.js, http-post-callback, null:256] SAML2 verify function executing {"ssoHostAddress":"management-vcenter.example.com","id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX"}
DEBUG [5b08949e26874c68] [services/authentication.js, http-post-callback, serializeUser:49] serializeUser() {"user":{"upn":"[email protected]","group":["vsphere.local\\Users","vsphere.local\\Administrators","vsphere.local\\CAAdmins","vsphere.local\\SystemConfiguration.BashShellAdministrators","vsphere.local\\SystemConfiguration.ReadOnly","vsphere.local\\SystemConfiguration.SupportUsers","vsphere.local\\SystemConfiguration.Administrators","vsphere.local\\LicenseService.Administrators","vsphere.local\\Everyone"],"nameID":"[email protected]","nameIDFormat":"http://schemas.xmlsoap.org/claims/UPN","sessionIndex":"_ABCDEFG"},"id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXX"}

ERROR [5b08949e26874c68] [services/errorHandling.js, http-post-callback, productionErrorRoute:106]
600.158: VError: Sending error response: 500 - "{\"errorCode\":\"IDENTITY_INTERNAL_SERVER_ERROR\",\"arguments\":[],\"message\":\"Identity Internal Server Error\",\"referenceToken\":\"ABC123\"}"

On the SDDC Manager you see entries similar to this in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log:

ERROR [common,5b08949e26874c68,a579] [c.v.v.s.client.impl.SoapBindingImpl,http-nio-127.0.0.1-7100-exec-1] SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
[...]
INFO  [common,5b08949e26874c68,a579] [c.v.v.s.c.i.SecurityTokenServiceImpl$RequestResponseProcessor,http-nio-127.0.0.1-7100-exec-1] Provided credentials are not valid.
ERROR [common,5b08949e26874c68,a579] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-7100-exec-1] Error while creating admin client using psc management-vcenter.example.com
com.vmware.evo.sddc.common.services.psc.exception.AuthenticationFailedException: Unable to obtain Security Token Service from SSO 'management-vcenter.example.com' as provided credentials are invalid
[...]
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

This can be caused by a mismatch in the password for the SSO administrator account, root account, or service account in VCF 5.2+

Resolution

SSH to SDDC Manager with vcf user and su to root
Run the below command to retrieve the SSO password.
```
lookup_passwords -u [email protected] -p '<SSO_PASSWORD>' -n 1 -s 10 -e PSC
```
1. If SSO password is not known then follow the Steps in KB Steps to recover passwords in SDDC Manager using local accounts when all certificates are expired and VC is not accessible to get the set password for the admin@local account
2. Retrieve the PSC / SSO password
```
lookup_passwords -u admin@local -p '<admin@local_password>' -n 1 -s 10 -e PSC
```
Log into vCenter and change the password for the SSO administrator account to match the password from Step #2

Restart services on the SDDC Manager

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh