How to replace a corrupted truststore in Aria Operations for Logs

Products

VMware Aria Suite

Issue/Introduction

Symptoms you might experience with a corrupted truststore:

Unable to successfully test pre-existing Aria Operations for Logs integrations (such as vIDM, vSphere, Aria Operations, and Active Directory).
Intermittent issues logging in with AD/SSO accounts
"Error loading certificates" is thrown accessing the Administration -> Certificates page
Exporting logs from Aria Operations for Logs fails with message '<HTML><head><title>Servlet Error | vRealize Log Insight </title><style> error-page-container
Exporting logs from Aria Operations for Logs to an NFS share fails with message "Failed to add new export task"
Navigating between UI tab yields the error "Failed to download fields. The list of fields is incomplete."
This issue can occur after upgrading to vRealize Log Insight 8.x, but is not exclusive to that task, and can occur outside of an upgrade.
The /storage/var/loginsight/runtime.log file contains entries similar to:

["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get alias of certificate. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]

["SslCertificateManagerScheduler-thread-1"/IP ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get custom CA certificates. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]

["pool-10-thread-6"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Failed to create socket factory]

java.security.KeyStoreException: problem accessing trust store

The /storage/var/loginsight/ui_runtime.log file contains entries similar to:

["https-jsse-nio2-443-exec-8"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.web.actions.APIProxyActionBean] [Error creating SSL socket factory.]
java.security.KeyStoreException: problem accessing trust store

The /storage/var/loginsight/plugins/vsphere/li-vsphere.log file contains entries similar to:

["pool-10-thread-2"/XXX.XXX.XXX.XXX INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Connecting to vCenter at https://vCenterIPorFQDN/sdk]

["pool-10-thread-2"/XXX.XXX.XXX.XXX INFO] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [verifySslCertificate is set to true, check for https certificates to be trusted.]

["pool-10-thread-2"/XXX.XXX.XXX.XXX ERROR] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Failed to Monitor VimEvents]
java.security.KeyStoreException: problem accessing trust store

Listing the contents of the truststore fails with an error
- To list the contents of the truststore:

Log in as root to the Aria Operations for Logs node in question via SSH
Note: If you do not know your root password, refer to How to reset the root password in Aria Operations for Logs
Run the following command
```
/usr/java/jre-vmware/bin/keytool -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass changeit
```
Note: The above command should return the certificates that exist in the truststore when run on a healthy node. If an error is received, please proceed to replace the truststore with the procedure outlined in this article

Environment

VMware Aria Operations for Logs 8.x
VMware vRealize Log Insight 8.x

Resolution

The preferred method to replace a corrupt truststore is by using a copy from a healthy Aria Operations for Logs node from the same cluster that does NOT return an error when listing the contents via the above command. If this is not available, the secondary method is to use the default truststore attached to this article (cacerts-nofips.tar.gz or cacerts-fips.tar.gz depending on whether FIPS is enabled in the cluster).

Non-FIPS enabled clusters

Perform one of the two operations:

Using a utility like WinSCP or FileZilla, copy a truststore from a working node in the same cluster, using it to replace the existing one in the following directory:

/usr/java/jre-vmware/lib/security/cacerts

Using a utility like WinSCP or FileZilla, copy the cacerts-nofips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:

SSH to the node as root
Extract the file
```
tar -xvzf /tmp/cacerts-no-fips.tar.gz
```

Replace the existing cacerts file with the new one and update ownership

mv /tmp/cacerts /usr/java/jre-vmware/lib/security/cacerts; chown root:root /usr/java/jre-vmware/lib/security/cacerts

Repeat steps 1-3 on all nodes which in the cluster that were determined to have corrupt truststores

FIPS enabled clusters

Perform one of the two operations:

Use a truststore from a working node in the same cluster to replace the existing one in the following directory:

/usr/java/jre-vmware/lib/security/cacerts

Using a utility like WinSCP or FileZilla, copy the cacerts-fips.tar.gz truststore attached to this article to the node's /tmp directory, extract it, and replace the existing one:

SSH to the node as root
Extract the file
```
tar -xvzf /tmp/cacerts-fips.tar.gz
```

Replace the existing cacerts file with the new one and update ownership

mv /tmp/cacerts /usr/java/jre-vmware/lib/security/cacerts; chown root:root /usr/java/jre-vmware/lib/security/cacerts

Repeat steps 1-3 on all nodes which in the cluster that were determined to have corrupt truststores

Additional Information

If you are unsure whether or not your cluster is FIPS enabled, and do not have access to the UI to verify:

Log in as root to any Aria Operations for Logs node in your cluster via SSH
Run the following command to determine the status of FIPS and if it is FIPS enabled.

/usr/lib/loginsight/application/sbin/fips.sh --all --status

FIPS mode check for all components.
Photon FIPS mode: activated
BouncyCastleFipsProvider in FIPS mode: on
BouncyCastelJsseProvider in FIPS mode: on
java.security keystore.type FIPS mode: on
java.security KeyManagerFactory.algorithm type FIPS mode: on
loginsight approved_only FIPS mode: activated
Apache Tomcat approved_only FIPS mode: activated
Apache Tomcat FIPS mode: on
Cassandra FIPS node: on
Internal config. fips-enabled = true

Attachments

cacerts-no-fips.tar get_app

cacerts-fips.tar get_app