Troubleshoot NSE Processing in 8.x

Products

• IT Management Suite
• Client Management Suite

Issue/Introduction

NSE's are not processing in a timely fashion, or need to move files out of EventQueue.

Note: This document covers queue performance in SMP 8.5 and higher. 

Module: w3wp.exe
Source: Altiris.NS.EventRouter.ProcessAgentHttpPostEvent
Description: Event not processed. IP: -.-.-.-; Error description: Altiris.NS.EventRouter+ServerBusyException: Queue was full.
at Altiris.NS.EventRouter.<>c__DisplayClass3.<RegisterDelivery>b__0(IDatabaseContext ctx)

Environment

ITMS 8.5 and newer

Resolution

First of all, if you desire to use Persistent Connections / Time Critical Management / and have enabled these features, please be advised that these features (specifically Persistent Connections) use a lot of CPU threads keeping connections opened on the SMP.  If you don't need Persistent Connections it's advised to turn this feature off. 

NOTE: If you are not using Time Critical Management, there is no need for Persistent Connections. 

If you want to use Time Critical Management with Persistent Connections, it's advised to make the following changes to the Core Settings in the Console (Settings > Notification Server > Core Settings).  These items below can be found in the Console under Settings > Notification Server > Core Settings by searching the Active Settings for "msgsthreadpoolsize"

It is suggested to make the following changes:

      MaxConcurrentDefaultMsgsThreadPoolSize   --> 4
      MaxConcurrentFastMsgsThreadPoolSize      --> 4
      MaxConcurrentLargeMsgsThreadPoolSize     --> 2
      MaxConcurrentPriorityMsgsThreadPoolSize  --> 4
      MaxConcurrentSlowMsgsThreadPoolSize      --> 2

NOTE: These settings are appropriate for an SMP with 32 CPU.  If more than 32 CPU's, these settings may be increased proportionally.

This information above regarding thread count changes, overrides the information suggested below when Persistent Connections are being used.

Follow these steps to continue troubleshooting NSE processing:

1. Stop the Altiris services:
   • Altiris Support Service
   • Altiris Client Message Dispatcher
   • Altiris File Receiver
   • Altiris Event Receiver
   • Altiris Service
   • Altiris Object Host Service
   • Altiris Service Hoster (if present)
   • Altiris Event Engine

    

2. Stop the IIS service
   • World Wide Web Publishing Service

    

3. Move NSEs out of the eventqueue and into a temporary folder (cut and paste the NSE files to a different location)
   • Default location: C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue
   • Or custom location that you may have (see Redirect EvtQueue folder to a drive other than C:\)

    

4.  (Optional) Consider increasing the thread counts and queue thresholds in the Core Settings to increase the threads available for each queue and balance the load across more queues. To make these changes: Open SMP Console and go to Settings>Notification Server>Core Settings. Under Notification Server Events (NSE)>NSE>Concurrent Messages and Queue directories, you should be able to find the following settings:

Note: The following values are given as an example and they are not the ideal values since each environment may need a different set of values. The general rule here is not to go too high at once, and in most cases, you may not need to change the entries the all.  More threads may lead to more Blocks in SQL.  Lower values may actually process faster.

< customSetting key="MaxConcurrentFastMsgsThreadPoolSize" type="local" value="20" /> (default is 16)
< customSetting key="MaxConcurrentSlowMsgsThreadPoolSize" type="local" value="10" /> (default is 4)

 

5. 8.5 RU1 or earlier only - drop foreign key

                          Note: Skip this step if on 8.5 RU2 or a newer release as the constraint has been removed by default and does not need to be re-created.

--Run this query first to drop foreign key (for versions older than 8.5RU2)

IF  EXISTS (SELECT * FROM sys.foreign_keys WHERE object_id = OBJECT_ID(N'[dbo].[FK_EventQueueEntryMetaData_EventQueueEntry]') AND parent_object_id = OBJECT_ID(N'[dbo].[EventQueueEntryMetaData]'))
ALTER TABLE [dbo].[EventQueueEntryMetaData] DROP CONSTRAINT [FK_EventQueueEntryMetaData_EventQueueEntry]
GO

 

       6.  Run the following queries to truncate these tables to clear the event and merge tables:

--truncate the following tables

truncate table eventqueue
truncate table eventqueueentry
truncate table [EventQueueEntryMetaData]
truncate table eventqueueprocess --only if it has stale data
truncate table eventqueuestatus --only if it has stale data


--validate that they are empty

select * from eventqueue
select * from eventqueueentry
select * from EventQueueEntryMetaData
select * from EventQueueProcess
select * from EventQueueStatus

 

7. Restart the SMP services:

• Altiris Support Service
• Altiris Client Message Dispatcher
• Altiris File Receiver
• Altiris Event Receiver
• Altiris Service
• Altiris Object Host Service
• Altiris Service Hoster (if present)
• Altiris Event Engine

8. Start the IIS service (World Wide Publishing Service)

9. Move the previously saved NSEs to the EvtInbox or delete them depending on retention requirements.

10. Add an exclusion rule, in your Antivirus software, for the .nse text files or the EvtQueue folder.

 

Additional Information

To perform additional troubleshooting of where NSEs are coming from see:
Evaluating NSE data using SQL when a deeper analysis is needed

Also see:

How to troubleshoot NSE processing issues
How to check for excessive NSE files in the EventQueue and be alerted when too high