The Log Insight service is unable to initiate the Cassandra service in Aria Operations for Logs

Products

• VMware Aria Suite

Issue/Introduction

The Loginsight service will be active / running but Cassandra will be down

nodetool-no-pass status

Failed to connect to '127.0.0.1:7199' - ConnectException: Connection refused)

The /var/log/vmware/loginsight/runtime.log shows entries similar to:

[2024-09-05 07:29:09.913+0000] ["vert.x-eventloop-thread-29"/Node IP INFO] [com.vmware.log.forwarder.verticle.HttpServerVerticle] [HTTP server running on port 5001]
[2024-09-05 07:29:09.913+0000] ["vert.x-eventloop-thread-29"/Node IP INFO] [com.vmware.log.forwarder.verticle.HttpServerVerticle] [HTTP server running on port 5001]
[2024-09-05 07:29:09.913+0000] ["vert.x-eventloop-thread-11"/Node IP INFO] [com.vmware.log.forwarder.verticle.LemansAgentVerticle] [Successfully started Lemans Agent verticle ######-11111-22222-33333]
[2024-09-05 07:29:09.913+0000] ["vert.x-eventloop-thread-9"/Node IP INFO] [com.vmware.log.forwarder.verticle.LemansAgentVerticle] [Successfully started Lemans Agent verticle ######-11111-22222-33333]
[2024-09-05 07:29:09.914+0000] ["vert.x-eventloop-thread-13"/Node IP INFO] [com.vmware.log.forwarder.verticle.LemansAgentVerticle] [Successfully started Lemans Agent verticle ######-11111-22222-33333]
[2024-09-05 07:29:09.914+0000] ["vert.x-eventloop-thread-29"/Node IPINFO] [com.vmware.log.forwarder.verticle.HttpServerVerticle] [HTTP server running on port 5001]
[2024-09-05 07:29:09.914+0000] ["vert.x-eventloop-thread-0"/Node IP INFO] [com.vmware.log.forwarder.verticle.LemansAgentVerticle] [Successfully started Lemans Agent verticle ######-11111-22222-33333]

Environment

VMware Aria Operations for Logs 8.12.x and later.

Cause

This issue arises because the cloud forwarder component continuously comes up repeatedly, preventing the Log Insight service from initiating the Cassandra service.

Resolution

NOTE: It's important to take snapshots of complete Aria Operations for logs cluster because proceeding with the below steps.

Steps to Remove Cloud Forwarder References

1. Log into the Primary Node: Access via SSH or Console as the root user

2. Open the Configuration File:
        Navigate to storage/core/loginsight/config/.
        Open loginsight-config.xml#number using a text editor (replace the number with the largest file number in that directory; use ls to check)

3. Remove Cloud Forwarder References:
        Find and delete the section referencing the cloud forwarder. 
        Example: 
             <cloud>
                   <channel name="Microsoft-Sentinel">
                   <cloud-url value="https://297043c4-f989" />
                   <cloud-key value="erwugfkjsdbwerjbjkge" />
                   </channel>
             </cloud>

4. Save and Close the File.

5. Reboot the Node: Execute the command: reboot -f

        6. Verify Changes after reboot, check that the log forwarded content is no longer in the loginsight-config.xml#number file

        7. On the other nodes, make sure the updated loginsight-config.xml#number is already in place and restart the node or Loginsight service.