Troubleshooting UI Access Issues in SDDC Manager

Products

• VMware Cloud Foundation

Issue/Introduction

• The SDDC Manager UI is inaccessible or stuck loading:

Environment

• VMware Cloud Foundation (VCF) 5.x
• VMware Cloud Foundation (VCF) 9.x

Cause

There are several possible causes:

• Disk space is full or constrained.
• The SSO password does not match.
• The vCenter root password does not match due to an out-of-band change.
• vCenter Server service accounts fail to authenticate or are disconnected.
• The SDDC Manager certificate is expired.
• SDDC Manager does not trust the Management vCenter root certificate.
• NTP fails to synchronize.
• Certificate files or certificate stores are corrupted.
• Database errors prevent access.
• Account permissions are misconfigured.
• Management vCenter host key validation fails.

Resolution

Follow these troubleshooting steps to identify the issue and implement a proper resolution.

1. Disk space is full or constrained:

1. SSH into the SDDC Manager, log in as the vcf user, and elevate to root using the following command: su root
2. Check the disk space by running the following command: df -h 
3. Check if any of the partitions on the SDDC Manager have over 80% usage.
4. Navigate (cd) into any directories that are full and safely clear out unnecessary files to free up space.
5. For additional guidance, refer to the KB article: SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up

2. The SSO password changed/does not match: 

• To identify and resolve this issue, refer to the following KB article: Logging into SDDC Manager fails with IDENTITY_INTERNAL_SERVER_ERROR

3.  The vCenter root password does not match due to an out-of-band change:

1. Run the following command in SDDC Manager to retrieve the password for the VCENTER resource (this will output the vCenter root password stored in SDDC Manager): lookup_passwords
2. Test if the stored password matches the actual password by initiating an SSH session to the vCenter Server directly from the SDDC Manager appliance.
3. If the SSH login fails, the root account on the vCenter Server is either locked out, or the password was changed out-of-band and no longer matches.
4. To unlock the account and reset the credentials, refer to the KB article: Reset Root Password.
   
   Note: SDDC requires SSH access to vCenter. Ensure that SSH is enabled on the Management vCenter (of the Management Domain).

4.  vCenter Server service accounts fail to authenticate or are disconnected:

1. Run the VCF Diagnostic Tool for SDDC Manager.
2. Inspect the CREDENTIAL CHECKS section of the output for any service account authentication failures.
3. If any failures are found, Retrieve the service accounts credentials from SDDC Manager.
4. Log in to the vSphere Client and update the passwords for those individual accounts on their respective vCenter Servers to match.

5. The SDDC Manager certificate is expired:

1. Run the VCF Diagnostic Tool (VDT) on the SDDC Manager appliance to check the current expiration status of the certificate.
2. If the tool indicates that the certificate has expired, refer to the following KB article for instructions on how to replace it: Reset SDDC Manager Certificate.

6. SDDC Manager does not trust the Management vCenter root certificate:

1. To verify that the SDDC Manager trusts the Management vCenter root certificate, refer to the following KB article: Import vCenter Root Cert into SDDC Manager Truststore.
2. Run the root synchronization script as outlined in the article.

7. NTP fails to synchronize:

1. Network Time Protocol (NTP) synchronization issues can cause severe slowness within the UI or prevent it from loading entirely.
2. Run the following commands to check the NTP status and restart the service:

1. 1. ntpq -p
   2. systemctl stop ntpd.service
   3. ntpdate ntp_server_ip
   4. systemctl start ntpd.service
   5. systemctl restart ntpd.service
   6. ntpq -p

8. Certificate files or certificate stores are corrupted:

1. If the root partition on the SDDC Manager reaches 100% capacity, changes to the commonsvcs trust store or the cacerts file may fail due to a lack of space, resulting in file corruption.
2. To verify if the commonsvcs trust store is corrupted, refer to the following KB article: SDDC manager UI failed load due to empty "trusted_certificates.store".
3. To verify if the cacerts file is corrupted, refer to the following KB article: SDDC manager UI failed to load due to corrupted cacerts file.

9. Database errors prevent access:

• Review the following KB articles for applicability:
  
  
  1. SDDC UI fails to load with error "Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: Request failed with status code 502".
  2. SDDC Manager UI is inaccessible. Error: 502 Bad Gateway due to the SDDC Database being down.
  3. SDDC Manager /data partition filling up to 100% frequently

10. Account permissions are misconfigured:

• To validate permissions, refer to the following KB article: SDDC manager UI is loading with 502 Bad default gateway error.

11. Management vCenter host key validation fails:

• To verify host key status, refer to the following KB article: Access to SDDC Manager UI is lost because of Management vCenter Server Key validation failed.

Additional Information

• If the vcf or root user accounts are locked on the SDDC Manager appliance, refer to KB How to reset SDDC Manager vcf and root user accounts to unlock and reset the credentials.
• If the SDDC Manager UI remains inaccessible after exhausting these troubleshooting steps, open a support case with Broadcom Support.