Aria Operations for Logs UI not accessible after removing nodes from the cluster

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• After removing a node (or nodes) from the Aria Operations for Logs cluster to go back to a one node cluster the primary node UI becomes inaccessible
• Attempts to start Cassandra on the primary node manually fails
• The /var/log/loginsight/runtime.log shows entries similar to the following when attempting to start services

[2025-10-01 11:31:51.671+0000] ["s9-admin-1"/XX.XX.XX.XX WARN] [com.datastax.oss.driver.internal.core.control.ControlConnection] [[s9] Authentication error (AuthenticationException: Authentication error on node XX.XX.XX.XX:9042: server replied with 'Unable to perform authentication: Cannot achieve consistency level LOCAL_QUORUM' to AuthResponse request)]

 

Environment

Aria Operations for Logs 8.18.x

Cause

When nodes are removed from a cluster, the replication factors in the Cassandra configuration are not adjusted and authentication to Cassandra fails

Resolution

To restore back to a one node cluster, perform the following on the primary node

***NOTE*** - Before performing the following steps ensure a snapshot/backup is taken for the Aria Operations for Logs primary node

1. SSH to the primary Aria Operations for Logs node as the root user and make sure that loginsight daemon isn't running:

systemctl stop loginsight

2. Update the "Authenticator" setting to allow any authentication (without requiring credentials) in /storage/core/loginsight/cidata/cassandra/config/cassandra.yaml file

Change the line

authenticator: PasswordAuthenticator

to

authenticator: AllowAllAuthenticator

3. Manually start Cassandra

/usr/lib/loginsight/application/sbin/li-cassandra.sh --startnow --force

4. Open cqlsh on the primary node

cqlsh-no-pass

5.  Alter the keyspace configurations to update the replication_factor (See screenshot below which have the correct settings for a one node cluster). 

Alter the replication with the following command for each keyspace with discrepancy from the screenshot which is the example for a standalone cluster:

 ALTER KEYSPACE "KeySpace Name" WITH replication = {'class': 'Strategy name', 'replication_factor' : 'No.Of replicas'};

6. After altering the necessary keyspaces, exit cqlsh by typing:

exit

7. Stop Cassandra

/usr/lib/loginsight/application/sbin/li-cassandra.sh --stopnow --force

8. Start the loginsight daemon service

systemctl start loginsight

Note that the "Authenticator" setting will be automatically reverted to its original value when starting loginsight daemon service