How to replace a corrupted truststore in VCF/Aria Operations for Logs

Products

VCF Operations/Automation (formerly VMware Aria Suite) VCF Operations

Issue/Introduction

Symptoms you might experience with a corrupted Truststore:

Unable to successfully test pre-existing Aria Operations for Logs integrations (such as vIDM, vSphere, Aria Operations, and Active Directory).
Intermittent issues logging in with AD/SSO accounts
"Error loading certificates" is thrown accessing the Administration > Certificates page
Exporting logs from Aria Operations for Logs fails with message '<HTML><head><title>Servlet Error | vRealize Log Insight </title><style> error-page-container'
Exporting logs from Aria Operations for Logs to an NFS share fails with message "Failed to add new export task"
Navigating between UI tab yields the error "Failed to download fields. The list of fields is incomplete."
The VC Collection Status shows ERROR "vSphere Collection Failed" under Integration > vSphere page in Operations for Logs.
One or more nodes shows 0 events are being ingested when checking Statistics page (To verify navigate to Management > System Monitor > Statistics > Events Ingestion Rate (Per Second))
A corrupted truststore can occur after upgrading to Aria Operations for Logs 8.x, but is not exclusive to that task, and can occur outside of an upgrade.

Log Entries that may be seen with a corrupted truststore

/storage/core/loginsight/runtime.log

ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get alias of certificate. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]

ERROR] [com.vmware.loginsight.database.dao.CACertificateDO] [Unable to get custom CA certificates. /usr/java/jre-vmware/lib/security/cacerts (No such file or directory)]

ERROR] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Failed to create socket factory]
 java.security.KeyStoreException: problem accessing trust store

ERROR] [com.vmware.loginsight.web.actions.APIProxyActionBean] [Error creating SSL socket factory.]
 java.security.KeyStoreException: problem accessing trust store

WARN] [com.vmware.loginsight.commons.RetryableOperation] [Operation failed: create-session-for-logdb. 2 attempts remaining. Retrying in 500 ms.]
com.vmware.loginsight.cassandra.CassandraException: Exception while getting session
	at com.vmware.loginsight.cassandra.CassandraUtil$2.call(CassandraUtil.java:409) ~[database-lib-li.jar:?]

/storage/core/loginsight/var/plugins/vsphere/li-vsphere.log

ERROR] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[https://<vCenter-FQDN/IP-Address>/sdk] Exception: 'HTTP transport error: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available' was thrown while trying to establish connection. Rethrowing exception.]

ERROR] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [ [<vCenter-FQDN/IP-Address>] Failed to Monitor VimEvents]
com.sun.xml.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available
  at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput (HttpClientTransport.java:102) ~[jaxws-rt-2.3.3.jar:2.3.3]

ERROR] [com.vmware.loginsight.vsphere.events.VimEventMonitor] [[XXX] Failed to Monitor VimEvents]
java.security.KeyStoreException: problem accessing trust store

Listing the contents of the Truststore fails with an error

To list the contents of the Truststore see the steps in the Resolution section below

Environment

VCF Operations for Logs 9.0.x
VMware Aria Operations for Logs 8.x

Cause

cacert (truststore) corruption in Operations for Logs often occurs during or after upgrades, abrupt system shutdowns, service crashes, or due to disk space exhaustion. It can also be caused by an improperly updated, empty, or corrupted /storage/core/loginsight/cidata/cassandra/config/cacert.pem file.

Resolution

Determine which node(s) have a corrupt truststore

Log in as root via SSH to all Operations for Logs nodes in the cluster
- Note:If you do not know your root password, refer to How to reset the root password in Aria Operations for Logs
(OPTIONAL) Determine if FIPS is enabled in the cluster (see Additional Information section below for more details)
```
/usr/lib/loginsight/application/sbin/fips.sh --all --status
```
List the certificates contained in the truststore
- FIPS not enabled:
```
/usr/java/jre-vmware/bin/keytool -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass 'changeit'
```
  Note: If the error keytool error: java.security.KeyStoreException: Unrecognized keystore format. Please load it with a specified type is seen, try the FIPS enabled command.
- FIPS enabled:
```
/usr/java/jre-vmware/bin/keytool -list -keystore /usr/java/jre-vmware/lib/security/cacerts -storepass 'changeit' -storetype bcfks -providerpath /usr/lib/loginsight/application/lib/lib/bc-fips-*.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
```
Note: The above command should return a list of all the certificates that exist in the truststore when run on a healthy node.
If an error is received other than Unrecognized keystore format, please proceed to replace the truststore with the procedure outlined in this article.

Replace a corrupted truststore file.

A corrupted truststore file must be replaced with a good copy.

The preferred source of a good copy is another healthy Operations for Logs node from the same cluster (Option 1).

If the truststore is corrupt on all nodes, the alternative method is to use the default truststore attached to this article (Option 2).

Option 1: Replace corrupted cacerts file using good copy from another node

Open the SSH session on the node with the corrupted cacerts file

SCP copy the truststore from the node with a good truststore file

scp root@:/usr/java/jre-vmware/lib/security/cacerts /usr/java/jre-vmware/lib/security/cacerts

Repeat steps 1 and 2 on all other nodes with corrupted truststores

Option 2: Replace corrupted cacerts file using good copy from this KB

Download the appropriate truststore file attached to the bottom of this KB
- FIPS not enabled: 'cacerts-no-fips.tar (0685G0000176bWoQAI_cacerts-no-fips.tar)'
- FIPS enabled: 'cacerts-fips.tar (0685G0000176bSsQAI_cacerts-fips.tar)'
Using a utility like WinSCP or FileZilla, copy the downloaded tar file to the /tmp directory on the node with corrupted truststore
Log in to the node with a corrupted truststore as root via SSH

Extract the truststore file

FIPS not enabled:

tar -xvzf /tmp/0685G0000176bWoQAI_cacerts-no-fips.tar

FIPS enabled:

tar -xvzf /tmp/0685G0000176bSsQAI_cacerts-fips.tar

Replace the existing truststore with the new one and update ownership

mv /tmp/cacerts /usr/java/jre-vmware/lib/security/cacerts; chown root:root /usr/java/jre-vmware/lib/security/cacerts

Repeat steps 2-5 on all other nodes with corrupted truststores

After replacing the truststore on a node, if the truststore continues to become corrupted, you may need to reboot that node after truststore replacement.

Additional Information

If you are unsure whether or not your cluster is FIPS enabled, and do not have access to the UI to verify:

Log in as root via SSH to any node in your cluster

Run the following command to determine the status of FIPS

/usr/lib/loginsight/application/sbin/fips.sh --all --status

Output when FIPS is enabled:

FIPS mode check for all components.
Photon FIPS mode: activated
BouncyCastleFipsProvider in FIPS mode: on
BouncyCastelJsseProvider in FIPS mode: on
java.security keystore.type FIPS mode: on
java.security KeyManagerFactory.algorithm type FIPS mode: on
loginsight approved_only FIPS mode: activated
Apache Tomcat approved_only FIPS mode: activated
Apache Tomcat FIPS mode: on
Cassandra FIPS node: on
Internal config. fips-enabled = true

Attachments

cacerts-no-fips.tar get_app

cacerts-fips.tar get_app