Error when trying to check the status of admin account password in Aria Operations for Logs: FAILED: Unable to get user data. Possible Cassandra is down.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Unable to login Web Interface of Aria Operations for Logs. "Error authenticating user"
• When trying to check the status of admin password, get the following error: FAILED: Unable to get user data. Possible Cassandra is down.
• While resetting the admin credentials, getting  following: ERROR: unable to get salt.
• After running through Reset and unlock the local admin account in VCF Operations for Logs, to check the admin status using the command:

root@hostname [ ~ ]# /usr/lib/loginsight/application/sbin/li-reset-admin-passwd.sh --checkAdminStatus
FAILED: Unable to get user data. Possible cassandra is down

• You receive this error when trying to access the Cassandra database on the Aria Operations for Logs node using this command:

root@hostname [ ~ ]# /usr/lib/loginsight/application/sbin/../lib/apache-cassandra-4.1.3/bin/cqlsh-no-pass
Connection error: ('Unable to connect to any servers', {'loopback:9042': PermissionError(1, "Tried connecting to [('loopback', 9042)]. Last error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)")})

Environment

VMware Aria Operations for Logs 8.x

Cause

The "reset admin password" script is failing because it is using cqlsh-no-pass command underneath, which is in-turn failing because of expired certificate.
The internal certificates are expiring, and it can be validated by executing the commands below:

keytool -list --keystore /usr/lib/loginsight/application/etc/truststore
keytool -list --keystore /usr/lib/loginsight/application/etc/3rd_config/keystore
echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

Resolution

Note: Emphasize to secure snapshots of all nodes (primary and worker nodes) before making any changes.

To resolve the issue, follow the steps below to reset admin password:

1. Edit the following file in all nodes: /storage/core/loginsight/cidata/cassandra/config/cqlshrc
2. In the last line: validate = true should be replaced with validate = false
3. After this change, run the li-reset-admin-passwd.sh script from this KB: Process to reset admin password

After the password has been changed and you are able to access the UI, you can replace the expired cert by following the steps mentioned below:

1. Login to primary node UI
2. Navigate to configuration
3. Click on SSL
4. Upload your certificate .pem file
5. Validate the expiry date of the external cert from the browser and internal cert using the command:

echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

Once the certificate is renewed, revert the changes done in file: /storage/core/loginsight/cidata/cassandra/config/cqlshrc.

1. Edit the following file: /storage/core/loginsight/cidata/cassandra/config/cqlshrc
2. In the last line: validate = false should be reverted to validate = true

Additional Information

To check if Cassandra is Up / Normal by running the command on primary node:

/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool-no-pass status

Guidance to replace certificates:

1. Install a custom SSL certificate
2. Generate a Self-Signed Certificate