java.security.KeyStoreException: Failed to load default trust store error in VCF Operations for Logs 9.x

Products

• VCF Operations

Issue/Introduction

Possible errors seen in logs:

• /storage/core/loginsight/var/runtime.log
  

  [YYYY-MM-DD HH:MM:SS.293+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to close file output stream:]java.io.EOFException: no data in keystore stream	

  [YYYY-MM-DD HH:MM:SS.300+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get alias of certificate. no data in keystore stream]

  [YYYY-MM-DD HH:MM:SS.300+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get custom CA certificates after checkAndRestoreTruststore was called. no data in keystore stream]

  [YYYY-MM-DD HH:MM:SS.302+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### ERROR] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Failed to create trust manager]java.security.KeyStoreException: Failed to load default trust store

For checking whether the truststore is corrupted or not on a node, the following command can help identify if truststore is corrupted :

keytool-no-provider -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit

If there is an error response after running this command, then the truststore is corrupted.

Environment

VCF Operations for Logs 9.0.x

Cause

Truststore corruption in VCF Operations for Logs often occurs during or after upgrades, abrupt system shutdowns, or disk space exhaustion.

Resolution

Replace corrupted truststore with backup copy:

Option 1:

1. Log in using root account via console or SSH
2. Verify corruption by running: 

    

   keytool-no-provider -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit

   If you receive an error this means the trustore is corrupted 

3. Issue cp command to replace the corrupted truststore with the backup copy. Use the appropriate command below for non-FIPS or FIPS enabled nodes respectively, see Additional Information section on how to determine if FIPS is enabled/disabled.
   
   Non-FIPS enabled node:
   

   cp /usr/lib/loginsight/application/etc/default_truststore/cacerts /usr/java/jre-vmware/lib/security

   
   FIPS enabled node:
   

   cp /usr/lib/loginsight/application/etc/default_truststore/FIPS/cacerts /usr/java/jre-vmware/lib/security

   Note: In case the FIPS directory is missing, you can copy the cacerts file from another VCF Operations for Logs node. The FIPS directory, and the cacerts file therein, is automatically created when FIPS is enabled.
4. Repeat steps on all nodes in the cluster where the trustore is corrupted

Option 2:

Using a utility like WinSCP or FileZilla, copy a truststore from a working node in the same cluster, using it to replace the existing one in the following directory:

/usr/java/jre-vmware/lib/security/cacerts

Additional Information

To verify if your cluster is FIPS enabled, and do not have access to the UI to verify:

1. Log in as root to any node in your cluster via SSH
2. Run the following command to determine the status of FIPS and if it is FIPS enabled.

3. /usr/lib/loginsight/application/sbin/fips.sh --all --status

   Expected output with FIPS disabled:

   FIPS mode check for all components.
   Photon require restart. Photon FIPS mode: deactivated
   BouncyCastleFipsProvider in FIPS mode: off
   BouncyCastelJsseProvider in FIPS mode: off
   java.security keystore.type FIPS mode: off
   java.security KeyManagerFactory.algorithm type FIPS mode: off
   loginsight approved_only FIPS mode: deactivated
   Apache Tomcat approved_only FIPS mode: deactivated
   Apache Tomcat FIPS mode: off
   Cassandra FIPS mode: off
   Internal config. fips-enabled = false