Aria Operations for Logs UI down after upgrade with error in logs "Provided username lisuper and/or password are incorrect"'

Products

• VMware Aria Suite

Issue/Introduction

• Upgraded Aria Operations for Logs the upgrade appears to complete but the UI is now inaccessible
• /storage/core/loginsight/var/runtime.log see the error Error from server: code=0100 [Bad credentials] message="Provided username lisuper and/or password are incorrect"')})
• Running the command /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/credentials-look-up on ssh session as root, we see that worker nodes had the username set to cassandra , this is incorrect the username should be lisuper. 
• Running the command on the primary node /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/cqlsh-no-pass on ssh session as root we received an error that the certificate is invalid

Environment

Aria Operations for logs 8.12 and later

Cause

Cassandra credentials where changed to a default Cassandra value

Resolution

1. Take snapshots of all nodes in the cluster without memory and without quiesce ensuring there valid backups of the cluster
   
   
2. Generate a self signed certificate as per KB 315949 
   
   
3. Log in to all nodes in the cluster as root via SSH

4. Stop vRLI Daemon service on all nodes
   
   service loginsight stop

5. On the primary node only, edit the cassandra.yaml file
   
   vi /storage/core/loginsight/cidata/cassandra/config/cassandra.yaml

6. Change line authenticator: PasswordAuthenticator
    to                authenticator: AllowAllAuthenticator

7. Start Cassandra on all nodes
   
   /usr/lib/loginsight/application/sbin/li-cassandra.sh --startnow --force

8. Make sure Cassandra is up (Status UN) on all nodes
   
   /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool-no-pass status

9. Start cqlsh (steps 9 to 15 are on the primary node only)
   
   /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/cqlsh-no-pass

10. Reset password for lisuper user:

    cqlsh> UPDATE system_auth.roles SET salted_hash = '$2a$10$1gMPBy9zSkDzKxdbU2v/gOslcMRPDcXVqmwQYBmi8MVgYvNdRZw/.' WHERE role = 'lisuper';
    cqlsh> exit

11. Generate new config file
    
    li-utility.sh --generate_new_config --force
     
    # example of output
    Running LI stop........done
    New config file generated: /storage/core/loginsight/config/loginsight-config.xml#104

12. Open the newly generated config file
    
    vi /storage/core/loginsight/config/loginsight-config.xml#<N>
    
    Note: Replace <N> with the new loginsight-config version number returned in step 10

13. Find lines referencing cassandra-user and cassandra-password

    Example
    <cassandra-user value="cassandra" />
<cassandra-password value="pass" />

14. Change cassandra-user value to lisuper
    
    <cassandra-user value="lisuper" />
    

15. Change cassandra-password value to cassandra
    
    <cassandra-password value="cassandra" />

16. Stop Cassandra on all nodes
    
    /usr/lib/loginsight/application/sbin/li-cassandra.sh --stopnow --force

17. Copy new config file into all other nodes

    scp /storage/core/loginsight/config/loginsight-config.xml#<N> 10.1.2.3:/storage/core/loginsight/config/loginsight-config.xml#<N>
    
    Note: Replace <N> with the new loginsight-config version number returned in step 10
              Replace 10.1.2.3 with the correct IP address for each node in the cluster

18. Start vRLI Daemon service on all nodes
    
    service loginsight start