Troubleshooting UI Access Issues in SDDC Manager

Products

VMware Cloud Foundation

Issue/Introduction

The SDDC Manager UI is inaccessible or stuck loading:

Alternatively the web page might show a yellow banner with the message:

Attempt ##/150: Unable to initialize the UI service due to "Status code 500: Unable to retrieve iDP Metadata: Failed to establish SSH session to <vCenter_FQDN> due to IO error.".

Environment

VMware Cloud Foundation (VCF) 5.x
VMware Cloud Foundation (VCF) 9.x

Cause

There are several potential causes for such issues:

Disk space is full or constrained.
The SSO password does not match.
The vCenter root password does not match due to an out-of-band change.
vCenter Server service accounts fail to authenticate or are disconnected.
The SDDC Manager certificate is expired.
SDDC Manager does not trust the Management vCenter root certificate.
NTP fails to synchronize.
Certificate files or certificate stores are corrupted.
Database errors prevent access.
Account permissions are misconfigured.
Management vCenter host key validation fails.

Resolution

Follow these troubleshooting steps to identify the issue and implement a proper resolution.

Disk space is full or constrained:
1. SSH into the SDDC Manager, log in as thevcf user, and elevate to root using the following command:
```
# su root
```
2. Check the disk space by running the following command:
```
# df -h
```
3. Check if any of the partitions on the SDDC Manager have more than 80% usage.
4. Using the cd command, navigate into any directories that are full and safely clear out unnecessary files to free up space.
5. For additional guidance, refer to the KB article SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up.
The SSO password changed/does not match:
- To identify and resolve this issue, refer to the article Logging into SDDC Manager fails with IDENTITY_INTERNAL_SERVER_ERROR.
The vCenter root password does not match due to an out-of-band change:
1. Run the following command in SDDC Manager to retrieve the password for the VCENTER resource (this will output the vCenter root password stored in SDDC Manager):
```
# lookup_passwords
```
2. Test if the stored password matches the actual password by initiating an SSH session directly from the SDDC Manager appliance to the vCenter.
3. If the SSH login fails, the root account on the vCenter Server is either locked out, or the password was changed out-of-band and no longer matches the one stored in SDDC Manager.
4. To unlock the account and reset the credentials, refer to the KB article Reset root password without rebooting - vCenter Server Appliance.
  Note: SDDC requires SSH access to vCenter. Ensure that SSH is enabled on the Management vCenter (of the Management Domain).
vCenter Server service accounts fail to authenticate or are disconnected:
1. Run the VCF Diagnostic Tool for SDDC Manager.
2. Inspect the CREDENTIAL CHECKS section of the output for any service account authentication failures.
3. If any failures are found, Retrieve the service accounts credentials from SDDC Manager.
4. Log in to vSphere Client and update the passwords for those individual accounts on their respective vCenter Servers to match.
The SDDC Manager certificate is expired:
1. Run the VCF Diagnostic Tool for SDDC Manager on the SDDC Manager appliance to check the current expiration status of the certificate.
2. If the tool indicates that the certificate has expired, refer to the SDDC Manager UI inaccessible or Regenerate button greyed out due to expired certificate in VMware Cloud Foundation for instructions on how to replace it.
SDDC Manager does not trust the Management vCenter root certificate:
1. To verify that the SDDC Manager trusts the Management vCenter root certificate, refer to the KB article Import vCenter Root Cert into SDDC Manager Truststore.
2. Run the root synchronization script as outlined in the article.
NTP fails to synchronize:
1. Network Time Protocol (NTP) synchronization issues can cause severe slowness within the UI or prevent it from loading entirely.
2. Run the following commands to check the NTP status and restart the service:
```
# ntpq -p
```
```
# systemctl stop ntpd.service
```
```
# ntpdate <ntp_server_ip>
```
```
# systemctl start ntpd.service
```
```
# systemctl restart ntpd.service
```
```
# ntpq -p
```
Certificate files or certificate stores are corrupted:
1. If the root partition on the SDDC Manager reaches 100% capacity, changes to the commonsvcs trust store or the cacerts file may fail due to a lack of space, resulting in file corruption.
2. To verify if the commonsvcs trust store is corrupted, refer to the KB article SDDC manager UI failed load due to empty "trusted_certificates.store".
3. To verify if the cacerts file is corrupted, refer to SDDC manager UI failed to load due to corrupted cacerts file.
Database errors prevent access:
Review the following KB articles for applicability:
Account permissions are misconfigured:
- To validate permissions, refer to the article: SDDC manager UI is loading with 502 Bad default gateway error.
Management vCenter host key validation fails:
- To verify host key status, refer to the following KB article: Access to SDDC Manager UI is lost because of Management vCenter Server Key validation failed.

Additional Information

If the vcf or root user accounts are locked on the SDDC Manager appliance, refer to the article Reset vcf and/or root user password expired in SDDC Manager - VMware Cloud Foundation. to unlock and reset the credentials.
Should the SDDC Manager UI remains inaccessible after exhausting the troubleshooting steps listed above, contact Broadcom support.