Logging into SDDC Manager fails with IDENTITY_INTERNAL_SERVER_ERROR
search cancel

Logging into SDDC Manager fails with IDENTITY_INTERNAL_SERVER_ERROR

book

Article ID: 327796

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

  • After authenticating, logging into SDDC Manager shows a message similar to: 
    {"message":"500 - \"{\\\"errorCode\\\":\\\"IDENTITY_INTERNAL_SERVER_ERROR\\\",\\\"arguments\\\":[],\\\"message\\\":\\\"Identity Internal Server Error\\\",\\\"referenceToken\\\":\\\"ABC123\\\"}\""}

     

  • On the SDDC Manager you see entries similar to this in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log: 
    VERBOSE [5b08949#####4c68] [services/sso-initialization.js, http-post-callback, null:256] SAML2 verify function executing {"ssoHostAddress":"management-vcenter.example.com","id":"########-####-####-####-##########"}
DEBUG [5b08949e#####c68] [services/authentication.js, http-post-callback, serializeUser:49] serializeUser() {"user":{"upn":"[email protected]","group":["vsphere.local\\Users","vsphere.local\\Administrators","vsphere.local\\CAAdmins","vsphere.local\\SystemConfiguration.BashShellAdministrators","vsphere.local\\SystemConfiguration.ReadOnly","vsphere.local\\SystemConfiguration.SupportUsers","vsphere.local\\SystemConfiguration.Administrators","vsphere.local\\LicenseService.Administrators","vsphere.local\\Everyone"],"nameID":"[email protected]","nameIDFormat":"http://schemas.xmlsoap.org/claims/UPN","sessionIndex":"_ABCDEFG"},"id":"########-####-####-####-##########"}

ERROR [5b08949e#####c68] [services/errorHandling.js, http-post-callback, productionErrorRoute:106]
600.158: VError: Sending error response: 500 - "{\"errorCode\":\"IDENTITY_INTERNAL_SERVER_ERROR\",\"arguments\":[],\"message\":\"Identity Internal Server Error\",\"referenceToken\":\"ABC123\"}"

     

  • On the SDDC Manager you see entries similar to this in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log: 
    ERROR [common,5b08949e#####c68,a579] [c.v.v.s.client.impl.SoapBindingImpl,http-nio-127.0.0.1-7##0-exec-1] SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
[...]
INFO  [common,5b0894#####74c68,a579] [c.v.v.s.c.i.SecurityTokenServiceImpl$RequestResponseProcessor,http-nio-127.0.0.1-7100-exec-1] Provided credentials are not valid.
ERROR [common,5b089#####874c68,a579] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-7100-exec-1] Error while creating admin client using psc management-vcenter.example.com
com.vmware.evo.sddc.common.services.psc.exception.AuthenticationFailedException: Unable to obtain Security Token Service from SSO 'management-vcenter.example.com' as provided credentials are invalid
[...]
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Cause

This can be caused by a mismatch in the password for the SSO administrator account, root account, or service account in VCF 5.2+

Resolution

  1. SSH to SDDC Manager with vcf user and su to root
  2. Run the below command to retrieve the SSO password. 
    lookup_passwords -u [email protected] -p '<SSO_PASSWORD>' -n 1 -s 10 -e PSC
    1. If SSO password is not known then follow the Steps in KB Steps to recover passwords in SDDC Manager using local accounts when all certificates are expired and VC is not accessible to get the set password for the admin@local account
    2. Retrieve the PSC / SSO password 
      lookup_passwords -u admin@local -p '<admin@local_password>' -n 1 -s 10 -e PSC
  3. Log into vCenter and change the password for the SSO administrator account to match the password from Step #2
  4. Restart services on the SDDC Manager 
    /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

 

In VCF 5.2+, the vCenter Service account password may become mismatched in the same manner. The steps below can be used to remediate a mismatched svc account password:

  1. SSH to SDDC Manager with vcf user and su to root
  2. Generate a token 
    TOKEN=$(curl -d '{"username" : "<SSO_ACCOUNT>", "password" : "<SSO_PASSWORD>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

Replace the appropriate account and password in the appropriate fields. For example:

TOKEN=$(curl -d '{"username" : "[email protected]", "password" : "VMware123!"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

To determine if the variable TOKEN above worked properly and is valid check it with this command:

echo $TOKEN

     

  3. Retrieve the passwords using: 
    curl -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | less
  4.  Locate the vCenter service account in the output (similar to 'svc-sddcname-vcentername').
  5. Log in to vCenter and set the service account password.
    1. Click Home.
    2. Click Administration.
    3. Click Single Sign-On > Users and Groups.
    4. Click the Users tab.
    5. Ensure the vsphere.local domain is selected.
    6. Click 'svc-sddcname-vcentername' (the service account name from step 3)
    7. Click Edit
    8. Set the password to the expected value from step 3 and click OK.
  6. Restart services on the SDDC Manager. 
    /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
Powered by Wolken Software