How to replace a corrupted truststore in Aria Operations for Logs

Products

VMware Aria Suite

Issue/Introduction

Symptoms you might experience with a corrupted Truststore:

Unable to successfully test pre-existing Aria Operations for Logs integrations (such as vIDM, vSphere, Aria Operations, and Active Directory).
Intermittent issues logging in with AD/SSO accounts
"Error loading certificates" is thrown accessing the Administration >> Certificates page
Exporting logs from Aria Operations for Logs fails with message '<HTML><head><title>Servlet Error | vRealize Log Insight </title><style> error-page-container'
Exporting logs from Aria Operations for Logs to an NFS share fails with message "Failed to add new export task"
Navigating between UI tab yields the error "Failed to download fields. The list of fields is incomplete."
The VC Collection Status shows ERROR "vSphere Collection Failed" under Integration >> vSphere page in Aria Operations for Logs.
One or more nodes shows 0 events are being ingested when checking Statistics page to verify navigate to Management > System Monitor > Statistics > Events Ingestion Rate (Per Second)

The /storage/core/loginsight/var/plugins/vsphere/li-vsphere.log file contains entries similar to:

ERROR] [com. vmware. loginsight.vsphere. events. VimEventMonitor] [[https://<vCenter-FQDN/IP-Address>/sdk] Exception: 'HTTP transport error: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available' was thrown while trying to establish connection. Rethrowing exception.]
["pool-9-thread-3"/#.#.#.# ERROR] [com. vmware. loginsight.vsphere.events. VimEventMonitor] [ [<vCenter-FQDN/IP-Address>] Failed to Monitor VimEvents]
com.sun.xml.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available
at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput (HttpClientTransport.java:102) ~[jaxws-rt-2.3.3.jar:2.3.3]
at com.sun.xml.ws. transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:193) ~[jaxws-rt-2.3.3.jar:2.3.3]
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest (HttpTransportPipe.java:115) ~[jaxws-rt-2.3.3.jar:2.3.3]  

["pool-10-thread-2"/XXX.XXX.XXX.XXX INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Connecting to vCenter at https://<vCenter-FQDN/IP-Address>/sdk]
["pool-10-thread-2"/XXX.XXX.XXX.XXX INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [verifySslCertificate is set to true, check for https certificates to be trusted.]
["pool-10-thread-2"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Failed to Monitor VimEvents]
java.security.KeyStoreException: problem accessing trust store

This issue can occur after upgrading to Aria Operations for Logs 8.x, but is not exclusive to that task, and can occur outside of an upgrade.

The /storage/core/loginsight/runtime.log file contains entries similar to:

["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get alias of certificate. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]
 ["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get custom CA certificates. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]
["pool-10-thread-6"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Failed to create socket factory]
 java.security.KeyStoreException: problem accessing trust store
 ["####-###-###-443-####-#"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.web.actions.APIProxyActionBean] [Error creating SSL socket factory.]
 java.security.KeyStoreException: problem accessing trust store

You may also see the following error in /storage/core/loginsight/runtime.log

[2025-08-13 08:38:07.230+0000] ["SslCertificateManagerScheduler-thread-1"/###.###.###.### WARN] [com.vmware.loginsight.commons.RetryableOperation] [Operation failed: create-session-for-logdb. 2 attempts remaining. Retrying in 500 ms.]
com.vmware.loginsight.cassandra.CassandraException: Exception while getting session
	at com.vmware.loginsight.cassandra.CassandraUtil$2.call(CassandraUtil.java:409) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraUtil$2.call(CassandraUtil.java:397) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.commons.RetryableOperation.run(RetryableOperation.java:98) ~[commons-lib.jar:?]
	at com.vmware.loginsight.cassandra.CassandraUtil.getSession(CassandraUtil.java:413) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraUtil.getPreparedStatement(CassandraUtil.java:524) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraClient.bindStatement(CassandraClient.java:389) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraDSL$Query.bind(CassandraDSL.java:459) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraDSL$Select.bind(CassandraDSL.java:745) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraDSL$ConstrainableQuery.splitSelectAllStatementByPartition(CassandraDSL.java:636) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraDSL$ConstrainableQuery.bindByPartition(CassandraDSL.java:667) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraDSL$Query.execAsync(CassandraDSL.java:432) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.cassandra.CassandraDSL$Query.exec(CassandraDSL.java:423) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.database.dao.CACertificateDAO.getCACertificateBlobs(CACertificateDAO.java:34) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.database.dao.CACertificateDO.getAllCACertificates(CACertificateDO.java:114) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.database.dao.CACertificateDO.checkAndUpdateTruststore(CACertificateDO.java:463) ~[database-lib-li.jar:?]
	at com.vmware.loginsight.daemon.shared.ssl.SslCertificateManager.checkAndUpdateTruststore(SslCertificateManager.java:830) ~[daemon-lib.jar:?]

Listing the contents of the Truststore fails with an error
- To list the contents of the Truststore:

1. Log in as root to the Aria Operations for Logs node in question via SSH
  Note: If you do not know your root password, refer to How to reset the root password in Aria Operations for Logs
2. Run the following command
```
/usr/java/jre-vmware/bin/keytool -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit
```
  Note: The above command should return the certificates that exist in the truststore when run on a healthy node. If an error is received, please proceed to replace the Truststore with the procedure outlined in this article

Environment

VMware Aria Operations for Logs 8.x

Resolution

The preferred method to replace a corrupt truststore is by using a copy from a healthy Aria Operations for Logs node from the same cluster that does NOT return an error when listing the contents via the above command. If this is not available, the secondary method is to use the default truststore attached to this article (cacerts-nofips.tar.gz or cacerts-fips.tar.gz depending on whether FIPS is enabled in the cluster). If after replacing the truststore on a corrupted node it continues to become corrupted, you may need to reboot that node after replacement.

Non-FIPS enabled clusters

Perform one of the two operations:

Using a utility like WinSCP or FileZilla, copy a truststore from a working node in the same cluster, using it to replace the existing one in the following directory:

/usr/java/jre-vmware/lib/security/cacerts

Using a utility like WinSCP or FileZilla, copy the cacerts-nofips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:

SSH to the node as root
Extract the file
```
tar -xvzf /tmp/cacerts-no-fips.tar.gz
```

Replace the existing cacerts file with the new one and update ownership

mv /tmp/cacerts /usr/java/jre-vmware/lib/security/cacerts; chown root:root /usr/java/jre-vmware/lib/security/cacerts

Repeat steps 1-3 on all nodes which in the cluster that were determined to have corrupt truststores

FIPS enabled clusters

Perform one of the two operations:

Use a truststore from a working node in the same cluster to replace the existing one in the following directory:

/usr/java/jre-vmware/lib/security/cacerts

Using a utility like WinSCP or FileZilla, copy the cacerts-fips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:

SSH to the node as root
Extract the file
```
tar -xvzf /tmp/cacerts-fips.tar.gz
```

Replace the existing cacerts file with the new one and update ownership

mv /tmp/cacerts /usr/java/jre-vmware/lib/security/cacerts; chown root:root /usr/java/jre-vmware/lib/security/cacerts

Repeat steps 1-3 on all nodes which in the cluster that were determined to have corrupt truststores

Additional Information

If you are unsure whether or not your cluster is FIPS enabled, and do not have access to the UI to verify:

Log in as root to any Aria Operations for Logs node in your cluster via SSH
Run the following command to determine the status of FIPS and if it is FIPS enabled.

/usr/lib/loginsight/application/sbin/fips.sh --all --status

FIPS mode check for all components.
Photon FIPS mode: activated
BouncyCastleFipsProvider in FIPS mode: on
BouncyCastelJsseProvider in FIPS mode: on
java.security keystore.type FIPS mode: on
java.security KeyManagerFactory.algorithm type FIPS mode: on
loginsight approved_only FIPS mode: activated
Apache Tomcat approved_only FIPS mode: activated
Apache Tomcat FIPS mode: on
Cassandra FIPS node: on
Internal config. fips-enabled = true

Attachments

cacerts-no-fips.tar get_app

cacerts-fips.tar get_app