Aria Operations for Logs forwarder events dropped due to target Pending queue full

Products

• VCF Operations

Issue/Introduction

• When forwarding logs from a Source Aria Operations for Logs (formerly Log Insight) cluster to a Target cluster, the source may intermittently trigger alerts indicating dropped events. This typically occurs when the ingestion pipeline on the target side cannot keep pace with the incoming data, causing the forwarder's memory and disk buffers to overflow.
• Email alerts received with the subject: Event Forwarder Events Dropped.
• Alert body contains: dropped XXXX events for forwarder target: <REDACTED_TARGET>, reason: Pending queue is full.
• runtime.log of source Aria Operations for Logs shows error below
  

["ImportingThread-4"/<REDACTED_IP> WARN] [com.vmware.loginsight.ingestion.forwarding.BaseForwarder] [Dropped 1000 events for target <REDACTED_TARGET>, reason: Pending queue is full. [169 suppressed]]
["ImportingThread-4"/<REDACTED_IP> WARN] [com.vmware.loginsight.ingestion.forwarding.BaseForwarder] [Dropped 90 events for target <REDACTED_TARGET>, reason: Pending queue is full. [1313 suppressed]]

• stats.log of the target node shows evidence of backpressure where parsing queues are overflowing to disk

pending-parsing-queue-disk-blocks:
  value = <HIGH_VALUE>

Environment

VMware Aria Operations for Logs 8.x

Cause

The root cause is a Disk I/O performance bottleneck on the Target Aria Operations for Logs nodes. When the target's virtual disks (vDisk) cannot acknowledge write operations fast enough, the ingestion parsing queue fills up. Once the target stops accepting new events or slows down significantly, the Source's "Pending Queue" (the memory buffer for forwarded events) reaches its limit, resulting in discarded logs.

Resolution

Since Aria Operations for Logs is a virtual appliance, validate the storage performance of the target nodes at the vSphere layer.

• Analyze Disk Latency via esxtop:

1. 1. Log in to the ESXi host running the target Log Insight VM via SSH and run esxtop.
   2. Press d to switch to the Disk view.
      GAVG/rd (Guest Average Read Latency): Should ideally be < 15ms.
      GAVG/wr (Guest Average Write Latency): Persistent spikes above 20ms indicate storage contention or slow underlying hardware.
      KAVG (Kernel Average Latency): High values suggest queuing within the ESXi storage stack.

• vCenter Performance Charts to monitor the target VM in vCenter:

1. 1. Navigate to Monitor > Performance > Advanced.
   2. Select Disk from the view dropdown.
   3. Review Average Write Latency and Disk Write Rate. Correlate spikes in latency with the "Pending queue is full" timestamps in the logs.

• Storage Infrastructure Validation

1. 1. Ensure the VM is not residing on a datastore with other high-latency workloads (e.g., heavily utilized databases).
   2. Verify that the underlying storage (vSAN, NFS, or VMFS) is healthy and not undergoing a rebuild or maintenance.
   3. Confirm the VM meets the IOPS requirements for its size (Small, Medium, Large, or Extra-Large) as per the official sizing guide.

Additional Information

For further instructions on bypassing storage latency within Aria Operations for Logs, please refer Event Drops in VMware Aria Operations for Logs due to Pending Queue Overload