Aria Operations for Logs UI running in FIPS mode is not accessible after SSL certificate expired

Products

• VMware Aria Suite

Issue/Introduction

• The Aria Operations for Logs cluster is FIPS enabled
• The SSL certificate has recently expired and/or been replaced for the Aria Operations for Logs cluster
• The /var/log/vmware/loginsight/runtime.log shows the following stack trace over and over again
   
  Caused by: java.security.cert.CertPathValidatorException: Could not validate certificate: certificate expired on 20250402133449GMT+00:00
        at org.bouncycastle.jcajce.provider.RFC3280CertPathUtilities.processCertA(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
        at org.bouncycastle.jcajce.provider.PKIXCertPathValidatorSpi.engineValidate(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
        at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.build(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
        at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.build(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
        at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
        at java.security.cert.CertPathBuilder.build(Unknown Source) ~[?:?]

 

• Running the below command on each cluster node, the certificate shows expired:

echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate 

• The Aria Operations for Logs UI does not come up properly and shows a start up screen like below

Environment

Aria Operations for Logs 8.18.x

Cause

The underlying issue is that thrift does not use the bcfks keystore when FIPS is enabled (but cassandra and tomcat both do), and the jks keystore does not get updated with new certs when FIPS is enabled, leading to thrift still using an expired certificate from the jks keystore

Resolution

To resolve this issue please perform the steps below:

1. Take snapshots of the Aria Operations for Logs cluster. See KB How to take a Snapshot of VMware Aria Operations for Logs
2. If not already done install a new CA signed or self-signed certificate for the Aria Operations for Logs cluster using the KB outlined here
3. After a new certificate has been installed on the cluster view and remove the current (expired certificate) from the non-FIPS keystore under the directory /usr/lib/loginsight/application/etc/3rd_config from all nodes of  Aria Operations for Logs cluster.

        NOTE - Retrieve the keystore password by running following command to successfully delete keystore.

grep keystore-password /storage/core/loginsight/config/loginsight-config.xml*

        NOTE- The keystore file may just be called keystore or may be called keystore.XXXXXX. Where XXXXXX equals a string of numbers.

       For keystore.XXXXXX

keytool -list -v -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.XXXXXX
keytool -delete -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.XXXXXX -alias loginsight

      And, for keystore

keytool -list -v -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore
keytool -delete -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -alias loginsight

    4. Copy the FIPS keystore with the newly replaced SSL certificate to the non-FIPS keystore with these commands.

      For keystore.XXXXXX

pw=$(grep 'syslog-ssl-keystore-password' $(ls -1 /storage/core/loginsight/config/loginsight-config* | tail -n 1) | cut -d\" -f2)

keytool -importkeystore -srckeystore /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks -srcstoretype bcfks -providerpath /usr/lib/loginsight/application/lib/lib/bc-fips-*.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -srcstorepass $pw -deststoretype jks -deststorepass $pw -destkeystore keystore.XXXXXX

      And for keystore

pw=$(grep 'syslog-ssl-keystore-password' $(ls -1 /storage/core/loginsight/config/loginsight-config* | tail -n 1) | cut -d\" -f2)

keytool -importkeystore -srckeystore /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks -srcstoretype bcfks -providerpath /usr/lib/loginsight/application/lib/lib/bc-fips-*.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -srcstorepass $pw -deststoretype jks -deststorepass $pw -destkeystore keystore

NOTE - You will need to run the commands above under the directory /usr/lib/loginsight/application/etc/3rd_config on all nodes of  Aria Operations for Logs cluster.

   5. Restart the loginsight service on all nodes, one at a time

systemctl restart loginsight