SDDC manager UI failed load due to empty "trusted_certificates.store"
search cancel

SDDC manager UI failed load due to empty "trusted_certificates.store"

book

Article ID: 316047

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

  • ​​​​SDDC Manager UI is not loading
  • Error in SDDC Manager /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log 
    caused by:
100.108: VError: Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: 502 - "<html>\n<head><title>502 Bad Gateway</title></head>\n<body>\n<center><h1>502 Bad Gateway</h1></center>\n<hr><center>nginx</center>\n</body>\n</html>
  • Trying to import certificates to SDDC Manager fails with below error 
    keytool error: java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted_certificates.store
java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted_certificates.store
    at java.base/sun.security.tools.keytool.Main.doCommands(Main.java:899)
    at java.base/sun.security.tools.keytool.Main.run(Main.java:415)
    at java.base/sun.security.tools.keytool.Main.main(Main.java:408)

Environment

  • VMware Cloud Foundation
  • VMware SDDC Manager

Cause

Corrupted SDDC Manager commonsvcs trust-store

Resolution

Rebuild the SDDC Manager Trusted Certificates store

Follow the below steps

  1. Take a snapshot of the SDDC Manager VM
  2. SSH to SDDC Manager VM using vcf account and su to root
  3. Backup the trusted_certificates.store and trusted_certificates.key files 
    cp /etc/vmware/vcf/commonsvcs/trusted_certificates.store /etc/vmware/vcf/commonsvcs/trusted_certificates.store.old
    cp /etc/vmware/vcf/commonsvcs/trusted_certificates.key /etc/vmware/vcf/commonsvcs/trusted_certificates.key.old

     

  4. Check if /storage/alt_root/etc/vmware/vcf/commonsvcs/trusted_certificates.store is available 
    find / -iname trusted_certificates.store | xargs ls -lh

     

    1. If available, run this command.  Omit this step if not available. 
      cp /storage/alt_root/etc/vmware/vcf/commonsvcs/trusted_certificates.store /etc/vmware/vcf/commonsvcs/trusted_certificates.store
    2. Run the below command to clear the trusted_certificates.store file 
      echo > /etc/vmware/vcf/commonsvcs/trusted_certificates.store

       

  5. Execute command ls -l /etc/vmware/vcf/commonsvcs and confirm the trusted_certificates.store file is empty by observing a "0" after "vcf".

    Sample output
    -r-------- 1 vcf_commonsvcs vcf 114 Feb 23 23:25 application.properties
-rw------- 1 vcf_commonsvcs vcf  8 Feb 20 23:11 java_trusted_certificates.key
-rw------- 1 vcf_commonsvcs vcf 4622 Feb 23 23:39 known_hosts
-rw------- 1 vcf_commonsvcs vcf  0 Feb 28 02:17 trusted_certificates.key
-rw------- 1 vcf_commonsvcs vcf  0 Feb 28 02:18 trusted_certificates.store

     

  6. Restart commonsvcs service 
    systemctl restart commonsvcs

     

  7. Insert the vCenter leaf and root certs back into SDDC manager.
    • For the vCenter leaf cert
      • From the sddc manager 
        scp root@<vcenter_serverfqdn>:/etc/vmware-vpx/ssl/rui.crt /tmp/vcenterleaf.cer
        pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key) && keytool -importcert -alias <aliasname> -file /tmp/vcenterleaf.cer -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass
      • For the vCenter root cert use this KB: How to import the vCenter root certificate into the SDDC manager TrustStore

  8. Restart SDDC Manager services 
    /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
Powered by Wolken Software