Troubleshoot NSE Processing in 8.5 and later

Products

• IT Management Suite
• Client Management Suite

Issue/Introduction

NSE's are not processing in a timely fashion, or need to move files out of EventQueue.

Note: This document covers queue performance in SMP 8.5 and higher. 

Module: w3wp.exe
Source: Altiris.NS.EventRouter.ProcessAgentHttpPostEvent
Description: Event not processed. IP: -.-.-.-; Error description: Altiris.NS.EventRouter+ServerBusyException: Queue was full.
at Altiris.NS.EventRouter.<>c__DisplayClass3.<RegisterDelivery>b__0(IDatabaseContext ctx)

Environment

ITMS 8.5 and later

Resolution

First of all, if Persistent Connections / Time Critical Management / Endpoint Management Workspaces / has been configured, please be advised that Persistent Connections uses a lot of CPU threads keeping connections opened on the SMP.  If you don't need Persistent Connections it's advised to turn them off.  If you want to use them, it's advised to make the following changes to the Core Settings in the Console (Settings > Notification Server > Core Settings).  These items will show in the Console if you search the Active Settings for "msgsthreadpoolsize"

Make the following changes:

      MaxConcurrentPriorityMsgsThreadPoolSize  --> 4
      MaxConcurrentFastMsgsThreadPoolSize      --> 4
      MaxConcurrentDefaultMsgsThreadPoolSize   --> 4
      MaxConcurrentLargeMsgsThreadPoolSize     --> 1
      MaxConcurrentSlowMsgsThreadPoolSize      --> 2

This information overrides the information suggested below if Persistent Connections are being used.

 

Follow these steps to continue troubleshooting NSE processing:

1. Stop the Altiris services:
   • Altiris Support Service
   • Altiris Client Message Dispatcher
   • Altiris File Receiver
   • Altiris Event Receiver
   • Altiris Service
   • Altiris Object Host Service
   • Altiris Service Hoster (if present)
   • Altiris Event Engine

    

2. Stop the IIS service
   • World Wide Web Publishing Service

    

3. Move NSEs out of the eventqueue and into a temporary folder (cut and paste the NSE files to a different location)
   • Default location: C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue
   • Or custom location that you may have (see KB 154355 for assistance in locating the folder if it is in a custom location)

    

4.  (Optional) Consider increasing the thread counts and queue thresholds in the "C:\ProgramData\Symantec\SMP\Settings\coresettings.config"  file to increase the threads available for each queue and balance the load across more queues. To make these changes: Open SMP Console and go to Settings>Notification Server>Core Settings. Under Notification Server Events (NSE)>NSE>Concurrent Messages and Queue directories, you should be able to find the following settings:

Note: The following values are given as an example and they are not the ideal values since each environment may need a different set of values. The general rule here is not to go too high at once, and in most cases, you may not need to change the entries the all.

< customSetting key="MaxConcurrentFastMsgsThreadPoolSize" type="local" value="20" /> (default is 16)
< customSetting key="MaxConcurrentSlowMsgsThreadPoolSize" type="local" value="10" /> (default is 4)

5. 8.5 RU1 or earlier only - drop foreign key

                          Note: Skip this step if on 8.5 RU2 or a newer release

--Run this query first to drop foreign key (for versions older than 8.5RU2)

 

IF  EXISTS (SELECT * FROM sys.foreign_keys WHERE object_id = OBJECT_ID(N'[dbo].[FK_EventQueueEntryMetaData_EventQueueEntry]') AND parent_object_id = OBJECT_ID(N'[dbo].[EventQueueEntryMetaData]'))

ALTER TABLE [dbo].[EventQueueEntryMetaData] DROP CONSTRAINT [FK_EventQueueEntryMetaData_EventQueueEntry]

GO

Note: In the 8.5 RU2 or older releases, the constraint has been removed by default and does not need to be re-created.

 

       6.  Run the following queries to truncate these tables to clear the event and merge tables:

--truncate the following tables

truncate table eventqueue

truncate table eventqueueentry

truncate table [EventQueueEntryMetaData]

truncate table eventqueueprocess --only if it has stale data

truncate table eventqueuestatus --only if it has stale data

 

--validate that they are empty

select * from eventqueue

select * from eventqueueentry

select * from EventQueueEntryMetaData

select * from EventQueueProcess

select * from EventQueueStatus

 

7. Restart the SMP services:

• Altiris Support Service
• Altiris Client Message Dispatcher
• Altiris File Receiver
• Altiris Event Receiver
• Altiris Service
• Altiris Object Host Service
• Altiris Service Hoster (if present)
• Altiris Event Engine

8. Start the IIS service (World Wide Publishing Service)

9. Move the previously saved NSEs to the EvtInbox or delete them depending on retention requirements.

10. Add an exclusion rule, in your Antivirus software, for the .nse text files or the EvtQueue folder.

 

Additional Information

To perform additional troubleshooting of where NSEs are coming from see:
Evaluating NSE data using SQL when a deeper analysis is needed

Also see:

How to troubleshoot NSE processing issues

How to check for excessive NSE files in the EventQueue and be alerted when too high

 

 

Old article title: Increasing Queue processing performance in SMP 8.5 and later