keytool error: java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted

search cancel

keytool error: java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted_certificates.store

book

Article ID: 316047

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

This KB defines the steps to rebuild / restore the SDDC Manager commonsvcs trust-store

Symptoms:

SDDC Manager UI is not launching
var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log shows the below error

caused by:
100.108: VError: Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: 502 - "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>

Trying to import certificates to SDDC Manager fails with below error

keytool error: java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted_certificates.store
java.lang.Exception: Keystore file exists, but is empty: /etc/vmware/vcf/commonsvcs/trusted_certificates.store
    at java.base/sun.security.tools.keytool.Main.doCommands(Main.java:899)
    at java.base/sun.security.tools.keytool.Main.run(Main.java:415)
    at java.base/sun.security.tools.keytool.Main.main(Main.java:408)

Environment

VMware Cloud foundation 5.x

Cause

Corrupted SDDC Manager commonsvcs trust-store

Resolution

Take snapshot of SDDC Manager VM
SSH to SDDC Manager VM using vcf account and su to root

Backup the trusted_certificates.store and trusted_certificates.key files

cp /etc/vmware/vcf/commonsvcs/trusted_certificates.store /etc/vmware/vcf/commonsvcs/trusted_certificates.store.old

cp /etc/vmware/vcf/commonsvcs/trusted_certificates.key /etc/vmware/vcf/commonsvcs/trusted_certificates.key.old

Check if /storage/alt_root/etc/vmware/vcdf/commonsvcs/trusted_certificates.store is available
- find / -iname trusted_certificates.store | xargs ls -lh
If so, run this command
- cp /storage/alt_root/etc/vmware/vcf/commonsvcs/trusted_certificates.store /etc/vmware/vcf/commonsvcs/trusted_certificates.store

Otherwise, run the following commands

Clear the trusted_certificates.store file by executing the below commands
- ```
> /etc/vmware/vcf/commonsvcs/trusted_certificates.store
```

Execute command 'ls -l /etc/vmware/vcf/commonsvcs' and confirm the trusted_certificates.store file is empty by observing a "0" after "vcf".

root@sddc-manager [ /etc/vmware/vcf/commonsvcs ]# ls -l /etc/vmware/vcf/commonsvcs
total 16
-r-------- 1 vcf_commonsvcs vcf 114 Feb 23 23:25 application.properties
-rw------- 1 vcf_commonsvcs vcf  8 Feb 20 23:11 java_trusted_certificates.key
-rw------- 1 vcf_commonsvcs vcf 4622 Feb 23 23:39 known_hosts
-rw------- 1 vcf_commonsvcs vcf  0 Feb 28 02:17 trusted_certificates.key
-rw------- 1 vcf_commonsvcs vcf  0 Feb 28 02:18 trusted_certificates.store

We'll then need to insert the vCenter leaf and root certs back into SDDC manager.
- For the vCenter leaf cert
  - From the sddc manager
    - scp root@<vcenter_serverfqdn>:/etc/vmware-vpx/ssl/rui.crt /tmp/vcenterleaf.cer
    - pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key) && keytool -importcert -alias <aliasname> -file /tmp/vcenterleaf.cer -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass
- For the vCenter root cert use this kb
  - https://knowledge.broadcom.com/external/article/316007/how-to-import-the-vcenter-root-certifica.html
Issue the following command to restart the SDDC Manager services:
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Feedback

thumb_up Yes

thumb_down No