logmon - detect keyword not found within a sliding window

Products

DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

Problem Statement

You need to monitor a log file for a specific keyword, and alert when they keyword has not been found within a certain time period.

The log file rolls over at a certain size and also when the application is restarted, so matching on a negative lookahead regex or using the "match on every run" option generates too many false positives, as logmon begins reading the newly generated log file and "forgets" that the keyword was already found.

Example Scenario

An administrator needs to monitor a log file (e.g., /var/log/example.log) for a specific completion keyword and alert if the keyword has not been seen in the log within the last 60 minutes.

Keyword: "Process Successfully Completed."

Check Frequency: Every 10 minutes

Trigger Condition: Raise a Critical alarm if the keyword fails to appear within a 60-minute rolling window.

Environment

DX UIM - Any Version
logmon probe - any version

Cause

By design, the DX UIM logmon probe parses text files line-by-line on a per-match basis. Using negative lookahead regular expressions directly on a log file will cause logmon to generate a false alarm for every single line that does not match the target string. "Match on every run" will similarly alert if the log file has rolled over, but the desired keyword has not yet appeared in the new log.

Resolution

To reliably monitor for the absence of an entry, use the Two-Profile State Strategy (Heartbeat Method):

Profile 1 (Tracker): Reads the log continuously and updates a state file timestamp whenever the keyword is logged.

Profile 2 (Checker): Runs on a schedule, executing a custom script every 10 minutes to verify if the state file timestamp is older than the allowed threshold.

Create the State Checker Script
Create a bash script on the monitored target server to evaluate the age of the state file and return an exit code based on the result. To do so, save the following script to a persistent directory (e.g., /opt/nimsoft/probes/system/logmon/check_absence.sh). This script takes three parameters, which allows it to be leveraged for multiple profiles without needing to duplicate the script itself:

1. The name of the state file to check
2. The expected maximum age (e.g. if the state file is older than this number of minutes, alert)
3. The string to return if the file exceeds the given maximum age

For example, to check a file called "uim_keyword_last_seen" and return "FILE_TOO_OLD" if the file is older than 60 minutes:

check_absence.sh "/opt/nimsoft/probes/system/logmon/uim_keyword_last_seen" 60 "FILE_TOO_OLD"

This will output "FILE_TOO_OLD" to stdout if the age of the given file is >60 minutes.
#!/bin/bash
STATE_FILE="$1"
MAX_AGE_MIN="$2"
ALERT_STRING="$3"


# Alert if state file was never created

if [ ! -f "$STATE_FILE" ]; then
    echo "$ALERT_STRING"
    exit 0
fi

# Age Check: Fire if file age is greater than MAX_AGE_MIN

MISSING=$(find "$STATE_FILE" -mmin +${MAX_AGE_MIN})

if [ -n "$MISSING" ]; then
    echo "$ALERT_STRING"
else
    echo "STATUS_OK"
fi
Configure Profile 1 (Keyword Tracker)

The first profile continuously watches the active log file and updates the state file's timestamp on match. The state file is held at /opt/nimsoft/probes/system/logmon/uim_keyword_last_seen and whenever this profile encounters the keyword in the log, it will update the timestamp on the state file. This means the timestamp on the state file will always reflect the last time the keyword was seen by the profile.

Profile Name: CheckForKeyword

Log file Path: /var/log/example,log

Mode: updates

Check Interval: 5 sec (default)

Disable "Generate Quality of Service" and "Generate Alarm"

Watcher Rule Name: Catch_Keyword_Success

Match Expression: /.*Process Successfully Completed.*/

Run Command On Match: touch /opt/nimsoft/probes/system/logmon/uim_keyword_last_seen

Configure Profile 2 (Alert Generator)

This profile executes the "check" script and triggers an alarm if the script returns an exit code indicating that the timestamp on the state file is older than 60 minutes.

Profile Name: AlertOnMissingKeyword

Mode: command

Command: /opt/nimsoft/probes/system/logmon/check_absence.sh "/opt/nimsoft/probes/system/logmon/uim_keyword_last_seen" 60 "FILE_TOO_OLD"

Check Interval: 10 minutes (adjust as per your requirements)

Watcher Rule Name: Keyword_Missing_Alert
Match Expression: /.*FILE_TOO_OLD.*/
Severity Level: Critical
Message to send on Match: "Expected keyword not seen in last 60 minutes."