Aria Operations for Logs UI crashes intermittently or is failing to start

Products

• VMware Aria Suite

Issue/Introduction

When loginsight service is restarted, the UI is accessible and crashes after a few minutes.  

 

Environment

Aria Operations for Logs 8.8

Aria Operations for Logs 8.10

Aria Operations for Logs 8.12

Aria Operations for Logs 8.14

 

Cause

When AD/LDAP certificate expired, and when FIPS is enabled, Bouncy Castle (bckfs keystore) crashes on expired cert.  

The following error has been logged in runtime.log:

[2023-03-03 14:29:17.483+0530] ["UncaughtExceptionHandler"/10.42.2.192 FATAL] [com.vmware.loginsight.daemon.StrataServiceFailureHandler] [Uncaught exception in thread: Thread[Thread-555,5,]; reason: Cannot write application data until initial handshake completed.]

java.lang.IllegalStateException: Cannot write application data until initial handshake completed.

        at org.bouncycastle.tls.TlsProtocol.writeApplicationData(Unknown Source) ~[bctls-fips-1.0.11.jar:1.0.11]

        at org.bouncycastle.jsse.provider.ProvSSLSocketDirect$AppDataOutput.write(Unknown Source) ~[bctls-fips-1.0.11.jar:1.0.11]

        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_321]

        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_321]

        at com.sun.jndi.ldap.Connection.cleanup(Connection.java:668) ~[?:1.8.0_321]

        at com.sun.jndi.ldap.Connection.run(Connection.java:1014) ~[?:1.8.0_321]

        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_321]

 

[2023-03-03 14:29:17.489+0530] ["CheckPerformer-thread-1"/10.42.2.192 WARN] [com.vmware.loginsight.aaa.ad.ActiveDirectoryValidator] [Unable to validate Active Directory credentials. Please check your Active Directory DNS name, port, and SSL settings as well as your username and password.; IllegalStateException: Cannot write application data until initial handshake completed.]

[2023-03-03 14:29:17.489+0530] ["CheckPerformer-thread-1"/10.42.2.192 WARN] [com.vmware.loginsight.prodcheck.lib.ActiveDirectoryCheck] [Wasn't able to authenticate to active directory]

com.vmware.loginsight.commons.exceptions.AuthenticationException: Unable to validate Active Directory credentials. Please check your Active Directory DNS name, port, and SSL settings as well as your username and password.

Resolution

The issue is resolved in Aria Operations for Logs 8.16: VMware Aria Operations for Logs 8.16 Release Notes

If an upgrade cannot be performed at this time the following steps may provide a workaround.

• SSH into the Primary node.
• Issue the command "service loginsight restart".  This will restart the loginsight service and enable the web UI access.
• Login as local admin to the Aria Operations for Logs UI.
• Go to "Authentication/Active Directory"
• Perform a "Test Connection"
• Trust and Save the new certificate.

Once the certificate has been saved, the new updated AD/LDAP certificate is now saved to the truststore.  The web UI will no longer be crashing.

It may happen that the loginsight service crashes too quickly and you are unable to login to the UI to accept the new certificate.

In this case perform a manual Operations for Logs upgrade to 8.16, you will still have to follow the steps above but the loginsight service will no longer crash.