SDDC Manager UI login fails with "Identity Internal Server Error:500"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• SDDC Manager initialization hang "VMWare Cloud Foundation is initializing..."

                   

• Login to SDDC Manager UI fails with the error:
  {"message":"Identity Internal Server Error","code":"IDENTITY_INTERNAL_SERVER_ERROR","status":500}
   
• Password health shows "Failed to get details".
  /opt/vmware/sddc-support/sos --password-health --domain-name ALL

SDDC: <SDDC_MANAGER>  |  vcf  |  <date>  | <date>  | GREEN
vCenter: <VCENTER>  |  root  |  <date>  | <date>  | Failed to get details


• The sddc_svc_account_sync.py script fails with the following traceback: psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 5432 failed: fe_sendauth: error sending password authentication
• Logs show credential issue:
  cat /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log |grep -i credentials

####-##-##T##:##:##.###+#### ERROR [common, ###################################, ####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,####] Cannot complete login due to incorrect credentials: <mgmt_vCenter>, <SVC_ACCOUNT>@vsphere.local

 

Environment

VMware Cloud Foundation (VCF) 5.x
VMware Cloud Foundation (VCF) 9.x

Cause

Credential mismatch between SDDC Manager and the Management vCenter Server/NSX, often caused by password drifts or service account sync failures.

Resolution

Steps to resolve the issue:

1. UI-Based Remediation (Recommended First Step): 
   a. Log in to the SDDC Manager UI.
   b. Navigate to Administration > Password Management.
   c. Select Remediate for the affected components (e.g., [email protected] for PSC or root for NSX-T Manager).
   d. Once remediated, perform a full restart of the management services to clear the cache: 
     /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

2. Scripted Workaround: 
   a. Download the sddc_svc_account_sync.py script and copy it to the /home/vcf/ directory on the SDDC Manager.
   b. Run the script as the root user: python /home/vcf/sddc_svc_account_sync.py.
   c. Follow the prompts to select the vCenter server and enter the SSO administrator credentials.

3. Manual Workaround (If steps 1 and 2 fail): 
   a. Obtain the service account secret from the SDDC Manager database using the provided curl command to retrieve the current password VCF is attempting to use.
   b. Update the password for the affected user in the vSphere Client (Administration > Single Sign On > Users and Groups).
   c. Restart the management stack as shown in Step 1d.

Additional Information

Reference: 

Retrieve the service accounts credentials from SDDC Manager

Documentation Link: SDDC Manager UI login fails with "Identity Internal Server Error:500"