Error: Unable to get user data. Possible Cassandra is down." when resetting or updating password in Aria Operations for logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When attempting to reset or unlock the admin account for Aria Operations for Logs using the script li-reset-admin-passwd.sh, the operation fails with the error message:
  

  Unable to get user data. Possible Cassandra is down.

• When running the nodetool-no-pass status command, it reports that the Cassandra service is UP and Normal for all nodes.
• When attempting to log in to the Cassandra database directly using the cqlsh-no-pass command, it fails with the following SSL error:

  Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': OSError(136, "Tried connecting to [('127.0.0.1', 9042)]. Last error: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4159)")})

• When checking the status of port 9042 with the command netstat -tulpn | grep LISTEN, it shows the port is set to LISTEN. The port 9042 set to LISTEN state implies that the service is running but communication is failing due to a missing or corrupt certificate.
• API commands will fail. Prior API integrations which worked previously, no longer work. Users will be unable to create a new bearer token.
• The following error in /storage/core/loginsight/var/runtime.log reports SSL handshake errors, indicating disrupted SSL communication between system components:

  ["netty-event-loop-28"/##.##.##.## ERROR] [play.core.server.netty.PlayRequestHandler] [Exception caught in Netty]
  io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
          at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
          at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]

Environment

• VMware Aria Operations for Logs 8.18.x

Cause

• The cacert.pem file located in the Cassandra configuration directory /storage/core/loginsight/cidata/cassandra/config/ is found to be empty or corrupted.
• There is an error in the custom-ssl-cerf script used during both Aria Suite Lifecycle (LCM) and manual self-signed certificate replacement workflows. Instead of copying the new certificate, the script incorrectly zeros out the existing /storage/core/loginsight/cidata/cassandra/config/cacert.pem file.
• 

Resolution

• This issue is planned to be fixed in a future release.
• To workaround this issue, perform the following steps:
  1. Access the Aria Operations for Logs node via SSH as the root user.
  2. Backup the cacert.pem file before any modification:

     cp -p /storage/core/loginsight/cidata/cassandra/config/cacert.pem /storage/core/loginsight/cidata/cassandra/config/cacert.pem.backup

  3. Copy the default cluster.pem certificate to the Cassandra configuration directory using the following command:

     cp /usr/lib/loginsight/application/etc/certs/cluster.pem /storage/core/loginsight/cidata/cassandra/config/cacert.pem

     Note: For a multi-node cluster, ensure this step is performed on all nodes.
  4. After replacing the cacert.pem file, validate and align the permissions of the file to 644 (-rw-r--r--) if not already:
     

     chmod 644 cacert.pem

  5. Run the li-reset-admin-passwd.sh script to reset or unlock the admin password.