VCF Logs UI fails to load after 9.x upgrade due to corrupted Java Keystore

Products

• VCF Operations

Issue/Introduction

• After a successful upgrade to VCF Operations for Logs 9.0.2, the user interface (UI) becomes inaccessible.
  
  
• The cluster is inaccessible via the Master Node FQDN, causing a failure in Fleet Life Cycle Management (LCM) workflows. This occurs because a mandatory log-related pre-check cannot establish a connection to the FQDN.
  
  
• While attempting to replace the corrupted keystores we see the error: keytool error: java.io.IOException: BCFKS KeyStore corrupted: MAC calculation failed.

Environment

VCF Operations for Logs 9.x

Cause

The keystore was corrupted.

Resolution

For a Multi Node Clusters follow the below steps.

1. Take a snapshot of all the nodes and log into the problematic node as root via SSH or Console.
2. Stop the loginsight service:
   

   service loginsight stop

3. Find the keystore password.
   

   grep keystore-password /storage/core/loginsight/config/loginsight-config.xml*

4. Run the below command to list the Keystore.
   

   keytool -list -v -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks

5. Steps to re-generate the keystore.bcfks
   

   keytool -list -v -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks

   pw=$(grep 'syslog-ssl-keystore-password' $(ls -1 /storage/core/loginsight/config/loginsight-config* | tail -n 1) | cut -d\" -f2)
   

   keytool-no-provider -importkeystore -srckeystore /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks -srcstoretype bcfks -providerpath /usr/lib/loginsight/application/lib/lib/bc-fips-*.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -srcstorepass $pw -deststoretype jks -deststorepass $pw -destkeystore keystore.bcfks

   keytool-no-provider -list -v -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks -storepass <paste the password from step-3>

   systemctl restart loginsight
   

Additional Information

For checking whether the truststore is corrupted or not on a node, the following command can help identify if truststore is corrupted :

keytool-no-provider -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit

 

To replace the corrupted truststore follow the KB: How to replace a corrupted truststore in VCF Operations for Logs