Upgrade hanging on an Aria Operations for Logs node

Products

• VMware Aria Suite

Issue/Introduction

When upgrading from 8.18.0 to 8.18.3, the node displays the new version on the Cluster page, but the Status keeps showing as Restarting, and the upgrade process doesn't go through.

This error can be seen in cassandra.log:

ERROR [MutationStage-3] 2025-03-06T15:45:11,965 StorageProxy.java:709 - Failed paxos prepare locally
java.lang.IllegalArgumentException: Out of range: 5400381800

Environment

Aria Operations for Logs upgrade from version 8.18.0 to 8.18.3.

Cause

The issue can be related to some corruptions or timestamp mismatches.
Because of this issue, the loginsight service fails to start on the node.

Resolution

Follow this procedure to resolve this issue:

1. Snapshot of the node(s) can be taken to be able to revert the change, however the pre-upgrade Snapshot is sufficient to be able go back to the pre-upgrade state if needed.
2. Log in as root onto the problematic node, and run these commands:
   mv /storage/core/loginsight/cidata/cassandra/data/system/paxos* /tmp/
mv /storage/core/loginsight/cidata/cassandra/data/system/_paxos* /tmp/
systemctl restart loginsight
3. If the cluster has other nodes pending upgrade, upgrade them manually (KB 344057 to proceed) and apply the same fix as above if the issue appears on any subsequent node.
4. Once the node/cluster is upgraded, verify that the product is up and running before removing the Snapshot(s).