How to replace a corrupted truststore in Aria Operations for Logs
search cancel

How to replace a corrupted truststore in Aria Operations for Logs

book

Article ID: 325769

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

This article details how to replace a corrupted truststore in vRealize Log Insight 8.x

Symptoms:
  • Unable to successfully test pre-existing vRealize Log Insight integrations (such as vIDM, vSphere, vRealize Operations, and Active Directory).
  • Intermittent issues logging in with AD/SSO accounts
  • "Error loading certificates" is thrown accessing the Administration -> Certificates page
  • Exporting logs from Log Insight fails with message '<HTML><head><title>Servlet Error | vRealize Log Insight </title><style> error-page-container
  • Exporting logs from Log Insight to an NFS share fails with message "Failed to add new export task"
  • Navigating between UI tab yields the error "Failed to download fields. The list of fields is incomplete."
     
  • This issue can occur after upgrading to vRealize Log Insight 8.x, but is not exclusive to that task, and can occur outside of an upgrade.
  • The /storage/var/loginsight/runtime.log file contains entries similar to:
[2023-10-03 07:15:15.307+0000] ["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get alias of certificate. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]

[2023-10-03 07:15:15.308+0000] ["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get custom CA certificates. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]

[2019-11-01 16:46:40.212+0000] ["pool-10-thread-6"/172.27.16.112 ERROR] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Failed to create socket factory]
java.security.KeyStoreException: problem accessing trust store
  • The /storage/var/loginsight/ui_runtime.log file contains entries similar to:
[2019-11-01 16:48:23.861+0000] ["https-jsse-nio2-443-exec-8"/172.27.16.112 ERROR] [com.vmware.loginsight.web.actions.APIProxyActionBean] [Error creating SSL socket factory.]
java.security.KeyStoreException: problem accessing trust store
  • The /storage/var/loginsight/plugins/vsphere/li-vsphere.log file contains entries similar to:
[2019-11-01 16:52:40.214+0000] ["pool-10-thread-2"/172.27.16.112 INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Connecting to vCenter at https://vCenterIPorFQDN/sdk]

[2019-11-01 16:52:40.214+0000] ["pool-10-thread-2"/172.27.16.112 INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [verifySslCertificate is set to true, check for https certificates to be trusted.]

[2019-11-01 16:52:40.215+0000] ["pool-10-thread-2"/172.27.16.112 ERROR] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Failed to Monitor VimEvents]
java.security.KeyStoreException: problem accessing trust store
  • Listing the contents of the truststore fails with an error
  • To list the contents of the truststore:
  1. Log in as root to the vRealize Log Insight node in question via SSH
Note: If you do not know your root password, refer to How to reset the root password in Aria Operations for Logs
  1. Run the following command /usr/java/jre-vmware/bin/keytool -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit

    Note: The above command should return the certificates that exist in the truststore when run on a healthy node.  If an error is received, please proceed to replace the truststore with the procedure outlined in this article


Environment

VMware Aria Operations for Logs 8.12.x
VMware vRealize Log Insight 8.x

Resolution

The preferred method to replace a corrupt truststore is by using a copy from a healthy vRealize Log Insight node from the same cluster that does NOT return an error when listing the contents via the above command.  If this is not available, the secondary method is  to use the default truststore attached to this article (cacerts-nofips.tar.gz or cacerts-fips.tar.gz depending on whether FIPS is enabled in the cluster).

Non-FIPS enabled clusters 

Perform one of the two operations:
  • Using a utility like WinSCP or FileZilla, copy a truststore from a working node in the same cluster, using it to replace the existing one in the following directory:
/usr/java/jre-vmware/lib/security/cacerts
  • Using a utility like WinSCP or FileZilla, copy the cacerts-nofips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:
  1. SSH to the node as root
  2. Extract the file
tar -xvzf /tmp/cacerts-no-fips.tar.gz
  1. Replace the existing cacerts file with the new one and update ownership
mv /tmp/cacerts /usr/java/jre-vmware/lib/security/cacerts; chown root:root /usr/java/jre-vmware/lib/security/cacerts
  1. Repeat steps 1-3 on all nodes which in the cluster that were determined to have corrupt truststores

FIPS enabled clusters

Perform one of the two operations:
  • Use a truststore from a working node in the same cluster to replace the existing one in the following directory:
/usr/java/jre-vmware/lib/security/cacerts
  • Using a utility like WinSCP or FileZilla, copy the cacerts-fips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:
  1. SSH to the node as root
  2. Extract the file
tar -xvzf /tmp/cacerts-fips.tar.gz
  1. Replace the existing cacerts file with the new one and update ownership
mv /tmp/cacerts /usr/java/jre-vmware/lib/security/cacerts; chown root:root /usr/java/jre-vmware/lib/security/cacerts
  1. Repeat steps 1-3 on all nodes which in the cluster that were determined to have corrupt truststores


    Additional Information

    If you are unsure whether or not your cluster is FIPS enabled, and do not have access to the UI to verify:
    1. Log in as root to any vRealize Log Insight node in your cluster via SSH
    2. Run the following command to determine the status of FIPS
    /usr/lib/loginsight/application/sbin/fips.sh --all --status


    Attachments

    cacerts-no-fips.tar get_app
    cacerts-fips.tar get_app