vRealize Log Insight not sending alerts or sending false positives

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• vRealize Log Insight alerts do not work correctly, either sending false positives alerts or not sending alerts when it should
• There are disk blocks in nodes for time-range when the alert query was run (in the UI under Administration > System Monitor > Statistics tab, Ingestion Queue table)
• If there are no disk blocks, then the existence of disk blocks in the past can be checked from the /storage/var/loginsight/runtime.log of your vRealize Log Insight nodes

Example: [2019-05-13 22:56:59.307+0000] ["PeerLogImporterService-thread-3445"/10.128.147.70 INFO] [com.vmware.loginsight.ingestion.queue.SizeCappedDiskBlock] [Create disk queue: parsing-queue.1579]

Cause

The alerts may not work correctly because the alert query does not consider events that exist in the disk blocks and are not indexed/stored yet. As a result, the alert condition hit/non-hit can be deceptive, because it is not based on all events for the specific time range.

Example: An alert is created based on some filter and the condition is "In the last 5 minutes, the number of matches found is less than 1."

If the events corresponding to the alert filter are in disk blocks when the alert query is run, then the query result will be 0, and false alert will be sent. Later, the events will be handled and stored and it will be obvious that there are results in that period of time and the alert should not be sent.

Resolution

• Analyze the cluster sizing to overcome constantly having disk blocks in environment.
• Increase the alert search period to minimize probability of false positive alerts if disk blocks are result of ingestion spikes and handled during a short period of time.

Additional Information

Sizing the vRealize Log Insight Virtual Appliance