[EventQueueDispatcher] [EventQueueDispatcher] thread failed.

book

Article ID: 194065

calendar_today

Updated On:

Products

Management Platform (Formerly known as Notification Server)

Issue/Introduction

The customer noticed this error in the NS logs multiple times during the day:

[EventQueueDispatcher] [EventQueueDispatcher] thread failed.

Value cannot be null.
Parameter name: path2
[System.ArgumentNullException @ mscorlib]
at System.IO.Path.Combine(String path1, String path2)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.EventQueue.Assign(String deliverPath, ParameterizedThreadStart worker, DispatchSlot slot, EventRow row)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.EventQueue.Pull(DispatchSlot slot, String deliverPath, ParameterizedThreadStart worker)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.EventQueue.Pull(String deliverPath, ParameterizedThreadStart worker)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.EventQueue.Push(String deliverPath, ParameterizedThreadStart worker, ICollection`1 rows)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.DistributeForQueue(Int32 queueId, ICollection`1 rows)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.LoadAndDistribute(EDistributeReason reason, Int32& completed)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.DistributeQueues(EDistributeReason reason, Int32& completed)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.DistributeQueues(EDistributeReason reason)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.DistributeStep(EDistributeReason reason)
at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformWork()
at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformWakeup()
at Altiris.Common.Threading.CommonThreadRunner.PerformWakeup()
at Altiris.NS.Threading.NsThreadRunner.PerformWakeup()
at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformLoop(Thread me, Int32 version, WaitHandle[] waiters, Int32 terminatingEvents)

Exception logged from:
at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.Log(Int32, String, Exception)
at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.HandleException(Exception)
at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformLoop(System.Threading.Thread, Int32, System.Threading.WaitHandle[], Int32)
at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.RunnerThreadProc()
at System.Threading.ThreadHelper.ThreadStart()

-----------------------------------------------------------------------------------------------------
Date: 6/24/2020 9:43:38 AM, Tick Count: 1923099515 (22.06:11:39.5150000), Size: 2.46 KB
Process: AeXSvc (33944), Thread ID: 231, Module: AeXSVC.exe
Priority: 1, Source: EventQueueDispatcher

Cause

Stale NSEs entry records in the database.

The customer at one point moved and deleted the incoming NSEs from the EvtQueue folder into a temp folder. Then he tried to re-process them but the NSE entries in the database didn't match anymore.

The code portion where it fails:

slot.ProcessFile = slot.FileInLine ? row.File : Path.Combine(deliverPath, row.File);

actually should not fail, as the file "is not inline" (as the Path.Combine() executed) i.e. - the "File" property should contain the correct NSE file name.
But the Combine() fails exactly for the file name argument (path2) being null.  Something was wrong either with NSE registration (that saved NULL into DB) or with "pull" logic not being able to read file name from the tables.

Environment

ITMS 8.5

Resolution

A simple test is to run the following query:

--For example, if I run this query about once every minute:

select min(id) as Oldest, max(id) as Newest
from EventQueueEntry

If you see that under the Oldest or Newest column the reference is not changing after refreshing the results a couple of times, it is most likely you have a NSE entry that is out of sync between what you have in the physical Queue folder and what the database is aware of.

In order to clear out those stale entries, please try the following:

  1. Stop the Altiris services:
    • Altiris Support Service
    • Altiris Service 
    • Altiris Client Message Dispatcher
    • Altiris File Receiver

     

  2. Stop the IIS service
    • World Wide Web Publishing Service (from the command line as administrator run IISRESET / STOP)

     

  3. Clearing the event and merge tables with this SQL:

    --Run first to drop foreign key

    /*Note: The following foreign key doesn't exist after 8.5 RU2 release, so if you have SMP 8.5 RU2 or later, you don't need to drop FK_EventQueueEntryMetaData_EventQueueEntry */

    IF  EXISTS (SELECT * FROM sys.foreign_keys WHERE object_id = OBJECT_ID(N'[dbo].[FK_EventQueueEntryMetaData_EventQueueEntry]') AND parent_object_id = OBJECT_ID(N'[dbo].[EventQueueEntryMetaData]'))

    ALTER TABLE [dbo].[EventQueueEntryMetaData] DROP CONSTRAINT [FK_EventQueueEntryMetaData_EventQueueEntry]

    GO

    Note: in 8.5 release, the constraint has been removed by default and does not need to be re-created.

     

    --truncate the following tables

    truncate table eventqueue

    truncate table eventqueueentry

    truncate table [EventQueueEntryMetaData]

    truncate table eventqueueprocess --only if it has stale data

    truncate table eventqueuestatus --only if it has stale data

     

     

    --validate that they are empty

    select * from eventqueue

    select * from eventqueueentry

    select * from EventQueueEntryMetaData

    select * from EventQueueProcess

    select * from EventQueueStatus



  4. Restart the SMP services:

    • Altiris Support Service
    • Altiris Service
    • Altiris Client Message Dispatcher
    • Altiris File Receiver

  5. Start the IIS service

    World Wide Web Publishing Service (from the command line as administrator run IISRESET /START)

 

Additional Information

KB 172741 "Increasing Queue processing performance in SMP 8.5"