How to replace a corrupted truststore in Aria Operations for Logs
book
Article ID: 325769
calendar_today
Updated On:
Products
VMware Aria Suite
Issue/Introduction
Symptoms you might experience with a corrupted truststore:
Unable to successfully test pre-existing vRealize Log Insight integrations (such as vIDM, vSphere, vRealize Operations, and Active Directory).
Intermittent issues logging in with AD/SSO accounts
"Error loading certificates" is thrown accessing the Administration -> Certificates page
Exporting logs from Log Insight fails with message '<HTML><head><title>Servlet Error | vRealize Log Insight </title><style> error-page-container
Exporting logs from Log Insight to an NFS share fails with message"Failed to add new export task"
Navigating between UI tab yields the error"Failed to download fields. The list of fields is incomplete."
This issue can occur after upgrading to vRealize Log Insight 8.x, but is not exclusive to that task, and can occur outside of an upgrade.
The /storage/var/loginsight/runtime.logfile contains entries similar to:
[2023-10-03 07:15:15.307+0000] ["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get alias of certificate. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]
[2023-10-03 07:15:15.308+0000] ["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get custom CA certificates. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]
[2019-11-01 16:46:40.212+0000] ["pool-10-thread-6"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Failed to create socket factory]
java.security.KeyStoreException: problem accessing trust store
The /storage/var/loginsight/ui_runtime.log file contains entries similar to:
[2019-11-01 16:48:23.861+0000] ["https-jsse-nio2-443-exec-8"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.web.actions.APIProxyActionBean] [Error creating SSL socket factory.] java.security.KeyStoreException: problem accessing trust store
The /storage/var/loginsight/plugins/vsphere/li-vsphere.logfile contains entries similar to:
[2019-11-01 16:52:40.214+0000] ["pool-10-thread-2"/XXX.XXX.XXX.XXX INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Connecting to vCenter at https://vCenterIPorFQDN/sdk]
[2019-11-01 16:52:40.214+0000] ["pool-10-thread-2"/XXX.XXX.XXX.XXX INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [verifySslCertificate is set to true, check for https certificates to be trusted.]
[2019-11-01 16:52:40.215+0000] ["pool-10-thread-2"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Failed to Monitor VimEvents] java.security.KeyStoreException: problem accessing trust store
Listing the contents of the truststore fails with an error
To list the contents of the truststore:
Log in as root to the vRealize Log Insight node in question via SSH
2. Run the following command /usr/java/jre-vmware/bin/keytool -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit
Note: The above command should return the certificates that exist in the truststore when run on a healthy node. If an error is received, please proceed to replace the truststore with the procedure outlined in this article
The preferred method to replace a corrupt truststore is by using a copy from a healthy vRealize Log Insight node from the same cluster that does NOT return an error when listing the contents via the above command. If this is not available, the secondary method is to use the default truststore attached to this article (cacerts-nofips.tar.gz or cacerts-fips.tar.gz depending on whether FIPS is enabled in the cluster).
Non-FIPS enabled clusters
Perform one of the two operations:
Using a utility like WinSCP or FileZilla, copy a truststore from a working node in the same cluster, using it to replace the existing one in the following directory:
/usr/java/jre-vmware/lib/security/cacerts
Using a utility like WinSCP or FileZilla, copy the cacerts-nofips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:
SSH to the node as root
Extract the file
tar -xvzf /tmp/cacerts-no-fips.tar.gz
Replace the existing cacerts file with the new one and update ownership
Repeat steps 1-3 on all nodes which in the cluster that were determined to have corrupt truststores
FIPS enabled clusters
Perform one of the two operations:
Use a truststore from a working node in the same cluster to replace the existing one in the following directory:
/usr/java/jre-vmware/lib/security/cacerts
Using a utility like WinSCP or FileZilla, copy the cacerts-fips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:
SSH to the node as root
Extract the file
tar -xvzf /tmp/cacerts-fips.tar.gz
Replace the existing cacerts file with the new one and update ownership