Admin login fails in VCF Operations for Logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When attempting to log into the VCF Operations for Logs (formerly Aria Operations for Logs / vRealize Log Insight) Web UI as the admin user, the login fails with the following error message on the screen:
  Unable to connect server.
  OR
  Cannot authenticate user
  
  
  
• The cassandra service crashes with the following log messages in /var/log/loginsight/cassandra.log:
  
  ERROR [HintsDispatcher:1] ####-##-## HH:MM:DD,SS HintsDispatchExecutor.java:243 - Failed to dispatch hints file ########-####-####-####-############-#############-#.hints: file is corrupted ({})"
  
  
• runtime.log shows connection failure to the node.
  
  [####-##-## HH:MM:SS.00094+0000] ["####-admin-1"/##.##.##.## WARN] [com.datastax.oss.driver.internal.core.control.ControlConnection] [[####] Error connecting to Node(endPoint=##.##.##.##:9042, hostId=null, hashCode=######), trying next node (AnnotatedConnectException: Connection refused: /##.##.##.##:9042)]

Environment

VCF Operations for Logs 9.0.x

Cause

This issue occurs when one or more Cassandra hints files become corrupted on a specific node.

The Cassandra database utilizes hints files to store data temporarily when a replica node is down or unreachable. If these files are corrupted (due to unexpected shutdowns, disk space exhaustion, or I/O errors), the Cassandra internal HintsDispatcher thread fails, preventing the Cassandra service from functioning correctly or communicating with other nodes over port 7000.

Resolution

To resolve this issue, you must locate the node with the corrupted hints files, safely remove them, and restart the Log Insight service.

Step 1: Identify the Affected Node

• Log into any working node via SSH as root.
• Check the cluster status by running the following command:

         # /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool-nopass status

         Identify which node is in a Down state (DN).

Step 2: Remove Corrupted Hints Files and Restart Service

 Please follow the instruction of Error "Failed to dispatch hints file" "file is corrupted" in cassandra.log

Additional Information

Error "Failed to dispatch hints file" "file is corrupted" in cassandra.log