Aria Operations for Log Insight node will not connect to cluster after 8.14 upgrade FIPS mode enabled
search cancel

Aria Operations for Log Insight node will not connect to cluster after 8.14 upgrade FIPS mode enabled

book

Article ID: 314175

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
- Upgraded Aria Operations for logs 8.12 environment to 8.14 (original release)
- Upgrade completed but shortly after several nodes went to a disconnected state.
- After rebooting primary node the disconnected nodes will connect briefly before returning to disconnected state.
- Ran 8.14 HF1 but no change - Ran 8.14.1 upgrade but no change
- Removed deleted nodes and redeployed. New nodes went to disconnected state after a few minutes.

Logs are flooded with following in runtime.log:
[2023-12-08 20:39:09.735+0000] ["PeerLogImporterService-thread-804"/xxx.xx.xx.xxx ERROR] [org.apache.t
hrift.server.TThreadPoolServer] [Thrift Error occurred during processing of message.]
org.apache.thrift.transport.TTransportException: org.bouncycastle.tls.TlsFatalAlert: certificate_unkno
wn(46)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178) ~[libthrif
t.jar:0.14.2]
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) ~[libthrift.jar:0.14.2]
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463) ~[libthrift.ja
r:0.14.2]
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361) ~[libthrift.ja
r:0.14.2]
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244) ~[lib
thrift.jar:0.14.2]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift.jar:0.14.2]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) [l
ibthrift.jar:0.14.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)

Environment

VMware Aria Operations for Logs 8.14.x

Resolution

Pending Resolution

Workaround:
In the logs of the unhealthy node is visible that there is unknown certificate problem.
Follow these steps:

Verify FIPS is enabled.
Stop loginsight service on unhealthy node

cd to /usr/lib/loginsight/application/etc
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/truststore .
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/truststore.bcfks .

cd 3rd_config
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/3rd_config/keystore.<epoch_time> .
scp root@<healthyNodeIP>:/usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks .
chmod 644 keystore.bcfks
chmod 644 keystore.<epoch_time>

Start Loginsight service