Patching Aria Operations for logs 8.10 to 8.12 fails.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Primary node is getting patched and is not able to proceed with the following worker nodes,It has in the screen of upgrade in-progress from more than 60 mins.
• The "Schema agreement not reached" error in Cassandra generally occurs when there is a mismatch in the schema between nodes in the cluster.  This issue can arise during a schema change, such as when a new column is added or a table is modified, and the nodes in the cluster aren't yet in sync. The runtime.logs shows this exception :"Schema agreement not reached after"

Environment

Aria Operations for Logs. 8.x

Resolution

• Revert snapshots without memory for all Aria for logs nodes prior the upgrade.
  
  
• Start Cassandra to Gradually Bring up the Nodes in Aria Ops for Logs Cluster

1.     Stop Aria Operations for logs Daemon service on all nodes:
       systemctl stop loginsight
      
2.     Make sure Aria Operations for logs Daemon Service and Cassandra are stopped 
       systemctl status loginsight
   
   
3.     Start Cassandra on each nodes
       /usr/lib/loginsight/application/sbin/li-cassandra.sh --startnow --force
          # output is similar to
             Running LI stop........done
           Starting Cassandra.....done
   
   
4.     Check that all nodes are up and run
       netstat -tulpn | grep 9042
   
   
5.    To check the status of Cassandra run the command below in Aria Operations for Logs primary node.
       nodetool-no-pass status
   
   
6.     If you see that some nodes are not UN status, stop/start cassandra on those nodes 
         /usr/lib/loginsight/application/sbin/li-cassandra.sh --stopnow --force
      /usr/lib/loginsight/application/sbin/li-cassandra.sh --startnow --force
          # output is similar to
          Running LI stop........done
      Starting Cassandra.....done

• Repair Cassandra DB

1. If all nodes are up (UN status) run repairing on all nodes
   /usr/lib/loginsight/application/lib/apache-cassandra-x.x.x/bin/nodetool flush
/usr/lib/loginsight/application/lib/apache-cassandra-x.x.x/bin/nodetool repair 
   
   
2. Once repair is over, stop Cassandra and start vRLI Daemon on each node. Make sure a node is up and running before proceeding to the next one:
   /usr/lib/loginsight/application/sbin/li-cassandra.sh --stopnow --force
systemctl start loginsight

• Create new snapshots without memory for all Aria for logs nodes prior try upgrade.
  
  
• Re-try upgrade.