Aria Operations for logs cluster is unreachable or takes very long time to respond

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Upon Successful Completion of certificate replacement task for Aria operations for logs cluster from Aria Suite Lifecycle, The VIP address is not responding sometimes and on the Management -> Cluster page one or more nodes show as 'Disconnected'
• When checking the /storage/core/loginsight/var/cassandra.log Errors like below can be found:

[play.core.server.netty.PlayRequestHandler] [Exception caught in Netty]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown


[com.vmware.loginsight.scheduled.metrics.ScheduledMetricCalculationService] [Failed to process metrics: ]
java.lang.RuntimeException: java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Environment

Aria Operations for logs 8.18.x

Aria suite lifecycle 8.18.0

Cause

Cluster certificate replacement from LCM did not propagate to a node. Other nodes cannot complete ssl handshake with disconnected node(s)

Resolution

1. SSH to all nodes and confirm the certificate validity by running the command:
   echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate 
   
   
2. To resolve the issue, follow the KB: Install a custom certificate in VMware Aria Operations for Logs 8.12 and Later