SDDC manager UI failed load due to empty "trusted

Products

VMware Cloud Foundation

Issue/Introduction

SDDC Manager UI is not loading

Error in var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

caused by:
100.108: VError: Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: 502 - "<html>\n<head><title>502 Bad Gateway</title></head>\n<body>\n<center><h1>502 Bad Gateway</h1></center>\n<hr><center>nginx</center>\n</body>\n</html>

Trying to import certificates to SDDC Manager fails with below error

keytool error: java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted_certificates.store
java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted_certificates.store
    at java.base/sun.security.tools.keytool.Main.doCommands(Main.java:899)
    at java.base/sun.security.tools.keytool.Main.run(Main.java:415)
    at java.base/sun.security.tools.keytool.Main.main(Main.java:408)

This KB defines the steps to rebuild / restore the SDDC Manager commonsvcs trust-store

Environment

VMware Cloud Foundation 5.x

Cause

Corrupted SDDC Manager commonsvcs trust-store

Resolution

Take a snapshot of the SDDC Manager VM
SSH to SDDC Manager VM using vcf account and su to root

Backup the trusted_certificates.store and trusted_certificates.key files

cp /etc/vmware/vcf/commonsvcs/trusted_certificates.store /etc/vmware/vcf/commonsvcs/trusted_certificates.store.old

cp /etc/vmware/vcf/commonsvcs/trusted_certificates.key /etc/vmware/vcf/commonsvcs/trusted_certificates.key.old

Check if /storage/alt_root/etc/vmware/vcf/commonsvcs/trusted_certificates.store is available

find / -iname trusted_certificates.store | xargs ls -lh

If available, run this command. Omit this step if not available.

cp /storage/alt_root/etc/vmware/vcf/commonsvcs/trusted_certificates.store /etc/vmware/vcf/commonsvcs/trusted_certificates.store

Run the below command to clear the trusted_certificates.store file
```
echo > /etc/vmware/vcf/commonsvcs/trusted_certificates.store
```

Execute command 'ls -l /etc/vmware/vcf/commonsvcs' and confirm the trusted_certificates.store file is empty by observing a "0" after "vcf".

root@sddc-manager [ /etc/vmware/vcf/commonsvcs ]# ls -l /etc/vmware/vcf/commonsvcs
total 16
-r-------- 1 vcf_commonsvcs vcf 114 Feb 23 23:25 application.properties
-rw------- 1 vcf_commonsvcs vcf  8 Feb 20 23:11 java_trusted_certificates.key
-rw------- 1 vcf_commonsvcs vcf 4622 Feb 23 23:39 known_hosts
-rw------- 1 vcf_commonsvcs vcf  0 Feb 28 02:17 trusted_certificates.key
-rw------- 1 vcf_commonsvcs vcf  0 Feb 28 02:18 trusted_certificates.store

Insert the vCenter leaf and root certs back into SDDC manager.

For the vCenter leaf cert

From the sddc manager

scp root@<vcenter_serverfqdn>:/etc/vmware-vpx/ssl/rui.crt /tmp/vcenterleaf.cer

pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key) && keytool -importcert -alias <aliasname> -file /tmp/vcenterleaf.cer -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass

For the vCenter root cert use this KB: How to import the vCenter root certificate into the SDDC manager TrustStore

Restart SDDC Manager services

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Additional Information

Case 36339658

The public cacerts store was corrupt.
/usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts
Fixed it by copying over the file from a working SDDC manager.