Queue was full error; missing inventory data (including Patch)

Products

• IT Management Suite

Issue/Introduction

The Event Queue on the Notification Server is full which prevents further event processing from clients.

 

Inventory data, including Basic Inventory, Inventory Solution, Inventory Pack for Servers, and Patch assessment scan inventory (Patch Compliance Reporting), is missing or not updating. 

"2/4/2015 9:06:17 AM","Event not processed. IP: xxx.xxx.xxx.xxx; Error: Failed to register NSE: C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\bc90093b285d4330a1a2b51f05924030.nse, source=5a6d46e7-e453-4931-8fb8-31f57ff4a4de, length=134.89 KB
Failed to register NSE: C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\bc90093b285d4330a1a2b51f05924030.nse, source=5a6d46e7-e453-4931-8fb8-31f57ff4a4de, length=134.89 KB
   [Altiris.NS.NseRegistrationFailedException @ Altiris.NS]
   at Altiris.NS.EventRouter.DeliverStream(Stream input, Guid source, Int32 priority, Guid metaDataType, String metaData)
   at Altiris.NS.EventRouter.DeliverPostedDataImpl(HttpRequest req)
   at Altiris.NS.EventRouter.ProcessAgentHttpPostEvent()

Queue was full
   [Altiris.NS.EventRouter+ServerBusyException @ Altiris.NS]
   at Altiris.NS.EventRouter.<>c__DisplayClass1.<RegisterDelivery>b__0(IDatabaseContext ac)
   at Altiris.Database.DatabaseContext`1.RetryAction(Int32 retries, Boolean transacted, Getter`1 getContext, Action`1 action, Action`1 retry, DeadlockRetryArgs& re, Exception& exception, Boolean inTransaction)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetryHelper(Int32 retries, Boolean inTransaction, Getter`1 getContext, Action`1 action, Action`1 retry, Boolean transacted)
   at Altiris.Database.DatabaseContext`1.PerformTransactedDeadlockRetry(Int32 retries, Getter`1 getContext, Action`1 action, Action`1 retry)
   at Altiris.NS.ContextManagement.AdminDatabaseContext.PerformTransactedDeadlockRetry(Action`1 action, String deadlockMessage, String category)
   at Altiris.NS.EventRouter.RegisterDelivery(Boolean fileInLine, String tempFile, Int64 fileSize, Guid source, Int32 priority, Guid metaDataType, String metaData)
   at Altiris.NS.EventRouter.DeliverStream(Stream input, Guid source, Int32 priority, Guid metaDataType, String metaData)

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception, String footer)
   at Altiris.NS.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)
   at Altiris.NS.EventRouter.ProcessAbstractException(Exception ex, HttpResponse& response, String sHostAddress)
   at Altiris.NS.EventRouter.ProcessAgentHttpPostEvent()
   at Altiris.Web.NS.Agent.PostEventHandler.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.5.3153.0&language=en&module=NC4rzHuqNsqbazgbQO/1Bk0bIOIxRY2RtqOC1icsXmnT1KD8HfWL6em5As29YpdOAC7HHMT5xlwpZNjMyFsdsf1NuHfMUnEcAqmeHWerf0g=&error=1500136771&build=**CEDUrlEnd**","EventRouter","w3wp.exe","306","Errors"
 

Environment

ITMS 7.x, 8.x

Cause

Communication issues caused all clients to queue massive amounts of events locally which were sent later to the Notification Server which flooded the queue with old events.

Resolution

Option1:
Allow Some time to NS to process all the events.

Option2:

• Delete All NSEs on NS "C:\ProgramData\Symantec\SMP\EventQueue\"
• Delete queued NSEs on clients by created a command script task with below value:
  FOR /F "tokens=2*" %%A IN ('REG.EXE QUERY "HKLM\Software\Altiris\Altiris Agent" /V "installdir"') DO SET AgentDir=%%B
  del /Q "%AgentDir%\Queue\%NSSERVER%\*.*"