Log Retention Discrepancy due to Archiving Failure in Aria Operations for Logs
search cancel

Log Retention Discrepancy due to Archiving Failure in Aria Operations for Logs

book

Article ID: 387145

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

  • This article describes the discrepancy between the oldest log timestamp displayed in the Index Partition and the Explore Logs in Aria Operations for Logs. The oldest log in the Index Partition shows a different date than what is shown in Explore Logs, causing confusion and concern.
  • Below screenshot from Index partition shows oldest logs is from Dec 8, 2024:

  • Below screenshot from explore logs shows, with same filter as mentioned in the Index partition, oldest log is from Nov 21,2024:

Environment

Aria Operations for logs 8.x

Cause

  • This issue occurs when the NFS archiving is failing due to disk quota issues, resulting in buckets getting stuck in "archiving state" and leading to inconsistency in log timestamps.
  • Runtime logs will indicate multiple instances of "Disk quota exceeded" errors and repeated warnings of failed bucket archiving attempts:

    /var/log/vmware/loginsight/runtime.log:

    [2025-01-02 22:12:27.163+0000] ["BucketManager-thread-1"/###.###.##.## WARN] [com.vmware.loginsight.ingestion.BucketManager] [Archiving bucket BucketId(uuid:#######-####-####-####-############, createTime:1735854481872) failed after 5 retries. Retiring it anyways.]
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    java.io.IOException: Disk quota exceeded
    [2025-01-02 22:31:07.251+0000] ["BucketManager-thread-1"/###.###.##.## WARN] [com.vmware.loginsight.ingestion.BucketManager] [Archiving bucket BucketId(uuid:#######-####-####-####-############, createTime:1735855602081) failed after 5 retries. Retiring it anyways.]

Resolution

Follow the below steps to remove the buckets that are stuck on archiving state from the Aria Operations for Logs inventory.

1. Reboot the Aria Operations for Logs node to clear the stuck buckets in the archiving state.
2. If rebooting does not resolve the issue, proceed with manually deleting the buckets that are stuck on archiving state:

a. Shut down vRLI Daemon service by running the command:
systemctl stop loginsight

b. Verify that the service is not started by running the command:
systemctl status loginsight


c. Identify a location on the appliance with sufficient space for the bucket file.

For example, use /tmp to represent the path with sufficient space for the remainder of the instructions. Delete the bucket altogether, or create a copy of the bucket file by running the command:
cp -r /storage/core/loginsight/cidata/store/<Bucket-ID> /tmp/

NOTE: Replace the Bucket-ID with the uuid value from the runtime.log 
For example:
cp -r /storage/core/loginsight/cidata/store/6a884d03-ec8e-44d9-8c36-ebfb283ebd6c /tmp/


d. Remove the buckets one by one from the bucket index by running the command:
cd /usr/lib/loginsight/application/sbin
./bucket-index <Bucket-ID> delete


For example:
cd /usr/lib/loginsight/application/sbin
./bucket-index 6a884d03-ec8e-44d9-8c36-ebfb283ebd6c delete


e. Start up the vRLI Daemon service by running the command:
systemctl start loginsight

Powered by Wolken Software