Notification Server Event (NSE) Backlog: Troubleshooting Delayed Reporting and Event Processing
search cancel

Notification Server Event (NSE) Backlog: Troubleshooting Delayed Reporting and Event Processing

book

Article ID: 421492

calendar_today

Updated On:

Products

IT Management Suite

Issue/Introduction

You may have noticed that your Symantec Management Agents (SMA) on client computers appear to be communicating with the Symantec Management Platform Server (SMP Server or Notification Server (NS)), but inventory data, policy updates, and task status reports (like software delivery results) are significantly delayed, often by several hours or days. The Symantec Management Console reports may show outdated information, and newly deployed tasks may take an unusually long time to initiate on the client endpoints.

Environment

ITMS 8.7.x, 8.8

Cause

The core issue is a backlog in the Notification Server Event (NSE) queue. Client computers send data (like basic inventory, policy status, and task completion reports) to the SMP Server as compressed XML files called NSEs. The SMP Server must process these events and write the data to the Symantec_CMDB database. When the rate of incoming events from thousands of agents exceeds the NS's processing capacity, the internal event queues fill up. This backlog causes all subsequent agent reports and events to be queued, resulting in the observed delays in reporting and task execution. The key to resolution is to identify the source of the high event volume and tune the event queue processing settings.

The usual root cause is a high volume of NSEs flooding the SMP Server, overwhelming the EventQueueDispatcher service and causing a resource bottleneck.

Probable Contributing Factors (The "Why"):

  1. Sudden Influx of Basic Inventory (NSE Flood): A mass event, such as a large number of agents checking in for the first time or a scheduled Basic Inventory policy for a large population running simultaneously, can generate an extreme number of NSEs in a short period.

  2. Misconfigured Client or Solution Policies: A policy (e.g., Inventory Solution, Patch Management) is configured to send data too frequently or is gathering an excessive amount of data, causing recurring high NSE volumes.

  3. Performance Bottleneck on the SMP Server or SQL Database: The server hosting the NS or the dedicated SQL database may be under-resourced (CPU, RAM, Disk I/O) and simply cannot process the NSEs quickly enough, regardless of the incoming volume.

Resolution

For more detailed information beyond this KB article, we strongly encourage you to review the following references and be familiar in how to use the available tools on this subject:

Using Event Data Analytics for understanding SMP Server performance
Allow/disallow new feature for NSE categorization processing
Core Performance: Connections on NS machine (TCP / IIS / WS)
Troubleshooting NSE processing issues
Evaluating NSE data using SQL when a deeper analysis is needed
How to check for excessive NSE files in the EventQueue and be alerted when too high

 

Follow these steps to diagnose the NSE backlog, identify the source, and apply recommended performance tuning.

Step 1: Diagnose the Backlog Using Console Reports

The most effective way to confirm and diagnose a backlog is by checking the new Notification Server Event Queue reports (available in recent ITMS 8.x versions).

  1. On the SMP Console, navigate to Reports > Notification Server Management > Server > Event Queue > Metadata Statistics.

  2. Run the Pending Events report.

    • Interpretation: This report shows the current count and total size of NSEs waiting to be processed. If the Total Pending NSE Count is consistently above 50,000–80,000, or if the queue is not clearing during off-peak hours, you have a confirmed backlog.

  3. Run the Processed Events Summary report.

    • Interpretation: Use this to drill down and see which Source or Product (e.g., Inventory Solution, Basic Inventory, or a specific task) is generating the highest volume of events over a defined period (e.g., the last 24 hours). This identifies the event type that is flooding the system.

Step 2: Log Analysis (Confirming Backlog)

If the reports are inaccessible or show an immediate issue, check the core NS log file for the [EventQueueDispatcher] entry.

  1. Path to Logs: Navigate to the Notification Server log directory:

    C:\ProgramData\Symantec\SMP\Logs\

  2. Open the most recent Notification Server log file (e.g., a50.log).

  3. Search for the [EventQueueDispatcher] entry, which tracks the pending NSEs.

    • Example Log Excerpt:

      [EventQueueDispatcher] [612.03 k / 2.89 GB] => [32 / 96 / 6.41 m @ 46(0) t, 53.1 i/s, 2.18:03:03]

    • How to Interpret Evidence: The first part, [612.03 k / 2.89 GB], represents the Total Pending NSEs (count / size). A count over 100k (100.00 k) is a critical indicator of a severe backlog that is actively impacting performance (Reference Troubleshooting NSE processing issues).

Step 3: Actionable Resolution Steps

Based on the findings from Step 1 and Step 2, apply the following steps in order.

A. Address the Overwhelming Source Event (Prioritize Reducing Influx)

If the Source is... Action to Take
Scheduled Basic Inventory 1. Stagger the Schedule: On the SMP Console, navigate to Settings > Agents/Plug-ins > Targeted Agent Settings > Symantec Management Agent Settings - All Windows Computers. Modify the Basic Inventory policy schedule to run on a staggered schedule (e.g., every 2 hours, start time randomized) to prevent all agents from reporting at the same time.
A Specific Solution Policy (e.g., Software Inventory)

1. Review Policy: Navigate to the specific policy (e.g., Manage > Policies > Software > Software Inventory > <Policy Name>). Review its schedule and Applied To targets.


2. Reduce Frequency/Scope: Reduce the collection frequency (e.g., from daily to weekly) or narrow the target of the policy.
No Clear Event Source Identified 1. Disable All Optional Inventory: Temporarily disable all non-essential inventory policies for a period (e.g., 2 hours). If the backlog immediately starts to clear, re-enable policies one by one to isolate the culprit.

B. Tune Event Queue Processing (Increase Processing Capacity)

If the server resources (CPU, RAM) are not maxed out, you can increase the system's ability to process NSEs by adjusting the core settings.

  1. Open the NS Configuration Utility: On the Notification Server, navigate to: C:\Program Files\Altiris\Notification Server\Bin\Tools and run NSConfigurator.exe.

  2. Adjust Thread Counts: Locate and increase the following setting values in the Core Settings tab.

    • EvtQueueMaxThreads: Increases the number of threads for event processing. Default is 16. Try increasing this to 24 or 32 (monitor CPU utilization after the change).

    • EvtQueueChunkSize: Defines how many events a single thread will process at a time. Default is 500. Try increasing this to 1000.

  3. Recycle the Service: Restart the Altiris Service to apply the configuration changes.

    • Validation: After restarting, continue monitoring the Pending Events report (Step 1) and the [EventQueueDispatcher] log entry (Step 2) to confirm that the pending count is now decreasing at a higher rate.

Note: Making these tuning changes will temporarily increase CPU and SQL resource usage, but it is necessary to eliminate the backlog.

Powered by Wolken Software