Adding a new node to Aria Operations for Logs cluster shows as disconnected

Products

• VMware Aria Suite

Issue/Introduction

• Newly deployed worker node shows as disconnected in cluster page shortly after adding to the cluster
• FIPS mode is enabled 
• One or more of the following files custom-key.pem , custom-cerf.der and also keystore.pkcs12 located at /usr/lib/loginsight/application/etc/certs/ are either missing or have empty contents 
• Applying VMware Aria Operations for Logs 8.14 New Worker Node is Disconnected Shortly After Addition to Cluster kb does not resolve the issue
• Error in the /storage/core/loginsight/var/cassandra.log is similar to 

  2025-04-04 14:28:51.749+0000] ["DaemonCommands-thread-45"/##.###.##.## ERROR] [org.apache.thrift.server.TThreadPoolServer] [Thrift Error occurred during processing of message.]
  org.apache.thrift.transport.TTransportException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
  Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
  Caused by: java.security.cert.CertificateException: Unable to construct a valid chain
  at org.bouncycastle.jsse.provider.ProvX509TrustManager.validateChain(ProvX509TrustManager.java:313) ~[bctls-fips-1.0.12.3.jar:1.0.12.3]
  Caused by: java.security.cert.CertPathBuilderException: TrustAnchor found but certificate validation failed.
  at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
  Caused by: java.security.SignatureException: certificate does not verify with supplied key
  at org.bouncycastle.jcajce.provider.X509CertificateObject.checkSignature(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]

• Error in the /storage/core/loginsight/var/runtime.log is similar to :

• [2025-04-04 14:41:25.695+0000] ["LogSearchWorker-thread-13"/##.###.##.## ERROR] [org.apache.thrift.server.TThreadPoolServer] [Thrift Error occurred during processing of message.]
  org.apache.thrift.transport.TTransportException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
  Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
  Caused by: java.security.cert.CertificateException: Unable to construct a valid chain
  Caused by: java.security.cert.CertPathBuilderException: TrustAnchor found but certificate validation failed.
  Caused by: java.security.SignatureException: certificate does not verify with supplied key

Environment

Aria Operations 8.18.x

Resolution

To resolve this issue please perform the steps below:

1. Take snapshots of the Aria Operations for Logs cluster. See KB How to take a Snapshot of VMware Aria Operations for Logs
2. If not already done install a new CA signed or self-signed certificate for the Aria Operations for Logs cluster using the KB outlined in the kb Install a custom certificate in VMware Aria Operations for Logs 8.12 and Later
3.  After a new certificate has been installed on the cluster view and remove the current (expired certificate) from the non-FIPS keystore under the below directory on the primary node /usr/lib/loginsight/application/etc/3rd_config
   • NOTE - The keystore file may just be called keystore or may be called keystore.XXXXXX. Where XXXXXX equals a string of numbers.
     • keytool -list -v -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.XXXXXX
     • keytool -delete -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.XXXXXX -alias loginsight
   • The keystore password can be found by running this command. Note down the value.
     • grep keystore-password /storage/core/loginsight/config/loginsight-config.xml*
4. Copy the FIPS keystore with the newly replaced SSL certificate to the non-FIPS keystore with these commands again run on the primary node
   • NOTE - You will need to run the commands below under the directory /usr/lib/loginsight/application/etc/3rd_config
     • pw=$(grep 'syslog-ssl-keystore-password' $(ls -1 /storage/core/loginsight/config/loginsight-config* | tail -n 1) | cut -d\" -f2)
     • keytool -importkeystore -srckeystore /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks -srcstoretype bcfks -providerpath /usr/lib/loginsight/application/lib/lib/bc-fips-*.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -srcstorepass $pw -deststoretype jks -deststorepass $pw -destkeystore keystore.XXXXXX
5. Restart the loginsight service on all nodes, one at a time
   • • systemctl restart loginsight

Additional Information

VMware Aria Operations for Logs 8.14 New Worker Node is Disconnected Shortly After Addition to Cluster

Aria Operations for Logs UI running in FIPS mode is not accessible after SSL certificate expired