Aria Operations for Log Insight node will not connect to cluster after 8.14 upgrade FIPS mode enabled

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:
- Upgraded Aria Operations for logs 8.12 environment to 8.14 (original release)
- Upgrade completed but shortly after several nodes went to a disconnected state.
- After rebooting primary node the disconnected nodes will connect briefly before returning to disconnected state.
- Ran 8.14 HF1 but no change - Ran 8.14.1 upgrade but no change
- Removed deleted nodes and redeployed. New nodes went to disconnected state after a few minutes.

Logs are flooded with following in runtime.log:
[2023-12-08 20:39:09.735+0000] ["PeerLogImporterService-thread-804"/xxx.xx.xx.xxx ERROR] [org.apache.t
hrift.server.TThreadPoolServer] [Thrift Error occurred during processing of message.]
org.apache.thrift.transport.TTransportException: org.bouncycastle.tls.TlsFatalAlert: certificate_unkno
wn(46)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178) ~[libthrif
t.jar:0.14.2]
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) ~[libthrift.jar:0.14.2]
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463) ~[libthrift.ja
r:0.14.2]
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361) ~[libthrift.ja
r:0.14.2]
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244) ~[lib
thrift.jar:0.14.2]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift.jar:0.14.2]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) [l
ibthrift.jar:0.14.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)

Environment

VMware Aria Operations for Logs 8.14.x

Resolution

Pending Resolution

Workaround:
In the logs of the unhealthy node is visible that there is unknown certificate problem.
Follow these steps:

Verify FIPS is enabled.
Stop loginsight service on unhealthy node

cd to /usr/lib/loginsight/application/etc
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/truststore .
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/truststore.bcfks .

cd 3rd_config
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/3rd_config/keystore.<epoch_time> .
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks .
chmod 644 keystore.bcfks
chmod 644 keystore.<epoch_time>

Start Loginsight service