UI is inaccessible via VIP or Primary node with 'Error Authenticating user' in Aria Operations for Logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Users receive the following error when accessing the UI via the VIP, 'Error Authenticating user'
• Verification of the admin account status results in the error, 'Cassandra is down' when running the command below;

  • /usr/lib/loginsight/application/sbin/li-reset-admin-passwd.sh --checkAdminStatus

• On one or more nodes in the cluster, there are errors in '/storage/core/loginsight/var/cassandra.log' stating there are corrupt hint files found;

  • ERROR [HintsDispatcher:1] YYYY-MM-DDTHH:MM:SS,SSS JVMStabilityInspector.java:## - Exception in thread Thread[HintsDispatcher:1,5,HintsDispatcher]
    org.apache.cassandra.io.FSReadError: java.io.IOException: Corrupt hint file found

• In some cases, comparing the output of 'df -h' shows that the '/storage/core' partition on different nodes have very different values.

Environment

VMware Aria Operations for Logs 8.18.x

Cause

Corrupt hint files and Cassandra corruption can be caused when storage becomes full.

Operations for logs requires that all nodes in a cluster are configured with the same volume of storage, as per document below;

• Increase the Storage Capacity of the VMware Aria Operations for Logs virtual appliance.

Resolution

1. Resolve any difference in storage capacity between nodes by adding disks of a relevant size to nodes that have less storage. The document below outlines this process;
   • Adding disks to Operations for Logs
2. Remove corrupt hints files from all nodes as per the KB below;
   • Error "Failed to dispatch hints file" "file is corrupted" in cassandra,log
3. Repair the cassandra database using the steps in the KB below;
   • UI is inaccessible as cassandra is down - Aria Operations for Logs