How to replace a corrupted truststore in VCF Operations for Logs

Products

• VCF Operations

Issue/Introduction

This article details how to replace a corrupted truststore in VCF Operations for Logs 9.0.x

Possible errors seen in logs:

• /storage/core/loginsight/var/runtime.log
  

  [YYYY-MM-DD HH:MM:SS.293+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to close file output stream:]java.io.EOFException: no data in keystore stream	

  [YYYY-MM-DD HH:MM:SS.300+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get alias of certificate. no data in keystore stream]

  [YYYY-MM-DD HH:MM:SS.300+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get custom CA certificates after checkAndRestoreTruststore was called. no data in keystore stream]

  [YYYY-MM-DD HH:MM:SS.302+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Failed to create trust manager]java.security.KeyStoreException: Failed to load default trust store

 

For checking whether the truststore is corrupted or not on a node, the following command can help identify if truststore is corrupted :

keytool-no-provider -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit

If there is an error response after running this command, then the truststore is corrupted.

Environment

VCF Operations for Logs 9.0.x

Cause

Truststore corruption in VCF Operations for Logs often occurs during or after upgrades, abrupt system shutdowns, or due to disk space exhaustion.

Resolution

Follow the procedure below to replace corrupted truststore with backup copy:

Option 1:

1. Log in using root account via console or SSH
2. Issue cp command to replace the corrupted truststore with the backup copy. Use the appropriate command below for non-FIPS or FIPS enabled nodes respectively, see Additional Information section on how to determine if FIPS is enabled/disabled.
   
   Non-FIPS enabled node:
   

   cp /usr/lib/loginsight/application/etc/default_truststore/cacerts /usr/java/jre-vmware/lib/security

   
   FIPS enabled node:
   

   cp /usr/lib/loginsight/application/etc/default_truststore/FIPS/cacerts /usr/java/jre-vmware/lib/security

    

3. Repeat steps on all nodes in the cluster that were determined to have corrupt truststores

Option 2:

Using a utility like WinSCP or FileZilla, copy a truststore from a working node in the same cluster, using it to replace the existing one in the following directory:

/usr/java/jre-vmware/lib/security/cacerts

Additional Information

If you are unsure whether or not your cluster is FIPS enabled, and do not have access to the UI to verify:

1. Log in as root to any VCF Operations for Logs node in your cluster via SSH
2. Run the following command to determine the status of FIPS and if it is FIPS enabled.

3. /usr/lib/loginsight/application/sbin/fips.sh --all --status

   Expected output with FIPS disabled:

   FIPS mode check for all components.
   Photon require restart. Photon FIPS mode: deactivated
   BouncyCastleFipsProvider in FIPS mode: off
   BouncyCastelJsseProvider in FIPS mode: off
   java.security keystore.type FIPS mode: off
   java.security KeyManagerFactory.algorithm type FIPS mode: off
   loginsight approved_only FIPS mode: deactivated
   Apache Tomcat approved_only FIPS mode: deactivated
   Apache Tomcat FIPS mode: off
   Cassandra FIPS mode: off
   Internal config. fips-enabled = false