Monitoring Linux D-State (Blocked) Processes using logmon

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

Customers need to monitor Linux devices for blocked processes (processes in the "D" or uninterruptible sleep state) and generate alarms in DX UIM when the count exceeds a threshold.

Environment

• DX Unified Infrastructure Management (all versions)
• logmon probe (v4.20 or higher recommended)
• Linux OS

Resolution

To monitor the count of D-state processes, use the logmon probe in command mode with a custom script or command string.

Step 1: Configure the Profile

1. Open the logmon probe configuration.
2. Create a new profile (e.g., Linux_DState_Monitor).
3. In the General tab:
   • Mode: select command.
   • Command: Enter the following string to count D-state processes: ps -eo stat --no-headers | awk '$1 ~ /^D/' | wc -l
   • Interval: Set to your desired polling frequency (e.g., 5 min).
   • Ensure Generate Alarm is selected.

Step 2: Define Watcher Rule and Variables

1. Navigate to the Watcher Rules tab and create a new rule.
2. In the Standard tab:
   • Match Expression: * 
3. In the Variables tab, create a new variable (e.g., DCount):
   • Source: Match Expression.
   • Match Group: 1.

Step 3: Configure Alarms and Thresholds

1. In the Variables tab for DCount, define the threshold:
   • Set the Expected value (e.g., < 15). The alarm will trigger if the value is outside this range.
2. In the Alarm tab of the Watcher Rule:
   • Severity: Select the desired level (e.g., Major).
   • Alarm Text: Number of D-state processes is $DCount, exceeding threshold.
   • Suppression Key: dStateCount (for correlation).

Additional Information

• Use the full path for commands if the probe cannot locate them in the environment path.
• Note that the 'Test' function in the probe GUI may return 'no result' even when the profile is working correctly; verify operation via the generated alarms or probe logs at level 3.
• It is recommended to use the most recent version of the logmon probe for consistent command execution.