Slow NSE processing. Unspecified Dataloader Exception encountered for event data class
search cancel

Slow NSE processing. Unspecified Dataloader Exception encountered for event data class

book

Article ID: 393885

calendar_today

Updated On:

Products

IT Management Suite

Issue/Introduction

You started to notice that NSEs were not processing as fast as they used to. Other processes at SQL query level seems to have "blocked", "locks", or "hung" queries. If you restart the SQL Server services, things starts to respond better for a while but the slowness comes back.

The following NS log entries may be present:

Entry 1:

Unspecified Dataloader SQL Exception encountered for inventory data class: 'AeX AC Client Status' (64b3c8ae-c9c2-48a4-8ded-486eb6bdfed3)

Database context does not have open connection.
   [DeadDatabaseContextException @ Altiris.Database.dll]
   at Altiris.Database.DatabaseContext<>.ProcessException(Exception)
   at Altiris.Database.RetryActionRequest<>.HandleRetryException(IDatabaseContext, IRetriedDatabaseContext, Exception)
   at Altiris.Database.RetryActionRequest<>.Perform(bool)
   at Altiris.Database.DatabaseContext<>.PerformWithDeadlockRetryHelper(int, int, bool, Getter<IDatabaseContext>, Action<IDatabaseContext>, Action<DeadlockRetryArgs>, bool, string, string)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.CommitData(IDataLoaderConnection)

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.NS.ContextManagement.AdminDatabaseContext.ReportException(string, string, Exception)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.CommitData(IDataLoaderConnection)
   at Altiris.NS.ResourceManagement.DataClassRowCollection+Loader.Load(DataClass, DataClassRows, bool, IDataLoaderConnection, bool)
   at Altiris.NS.AgentManagement.AgentStatusCollector.AgentStatusCollector.FlushStatusUpdate()
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformWakeup()
   at Altiris.Common.Threading.CommonThreadRunner.PerformWakeup()
   at Altiris.NS.Threading.NsThreadRunner.PerformWakeup()
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformLoop(Thread, int, WaitHandle[], int)
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.RunnerThreadProc()
   at System.Threading.ThreadHelper.ThreadStart()

-----------------------------------------------------------------------------------------------------
Date: 4/5/2025 11:18:57 PM, Tick Count: 12187156 (03:23:07.1560000), Size: 2.03 KB
Process: w3wp (33660), Thread ID: 88, Module: Altiris.Database.dll
Priority: 2, Source: DataLoader

 

Entry 2:

Rollback during retry was failed: type=[None], ctxBeforeRollback=[[2/89/0] {Tran, TranOwner, ReadCommitted, Closed} id=23203, t='None' ([e3ea4649368d4112be1d3c1f1620aaea]), s=[#2/89] id=5, AdminDatabaseContext], ctxAfterRollback=[[2/89/0] {TranOwner, ReadCommitted, Closed} id=23203, t='Rolledback' ([e3ea4649368d4112be1d3c1f1620aaea]), s=[#2/89] id=5, AdminDatabaseContext], possible cause:

ExecuteNonQuery requires an open and available Connection. The connection's current state is closed.
   [InvalidOperationException @ System.Data.dll]
   at System.Data.SqlClient.SqlCommand.ValidateCommand(string, bool)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource<object>, string, bool, int, out bool, bool, bool)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Altiris.Database.DatabaseContext<>.ExecuteNonQuery(string, object[])
   at Altiris.Database.DatabaseContext<>.Rollback(bool)
   at Altiris.NS.ResourceManagement.Database.SqlClientConnectionBase.RollbackTransaction()
   at Altiris.NS.ResourceManagement.DataClassRowCollection+Loader.Update(DataClass, DataClassRows, NullableDictionary<Guid,ResourceDLInfo>, bool, Dictionary<Guid,Guid>, IDataLoaderConnection)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.TryCommit(IDataLoaderConnection, bool, ICollection<Guid>, ICollection<Guid>)
   at Altiris.NS.ResourceManagement.DataClassRowCollection+<>c__DisplayClass25_0.<Commit>b__0(IDatabaseContext)
   at Altiris.NS.ContextManagement.Altiris.NS.ContextManagement.AdminDatabaseContext+<>c__DisplayClass28_0<>.<PerformTransactedDeadlockRetry>b__0(IDatabaseContext)
   at Altiris.Database.RetryActionRequest<>.Perform(bool)

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.Database.RetryActionRequest<>.HandleRetryException(IDatabaseContext, IRetriedDatabaseContext, Exception)
   at Altiris.Database.RetryActionRequest<>.Perform(bool)
   at Altiris.Database.DatabaseContext<>.PerformWithDeadlockRetryHelper(int, int, bool, Getter<IDatabaseContext>, Action<IDatabaseContext>, Action<DeadlockRetryArgs>, bool, string, string)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.Commit(IDataLoaderConnection)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.OnCommitHandler(DCCommitEventArgs)
   at Altiris.Database.RetryActionRequest<>.Perform(bool)
   at Altiris.Database.DatabaseContext<>.PerformWithDeadlockRetryHelper(int, int, bool, Getter<IDatabaseContext>, Action<IDatabaseContext>, Action<DeadlockRetryArgs>, bool, string, string)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.CommitData(IDataLoaderConnection)
   at Altiris.NS.ResourceManagement.DataClassRowCollection+Loader.Load(DataClass, DataClassRows, bool, IDataLoaderConnection, bool)
   at Altiris.NS.AgentManagement.AgentStatusCollector.AgentStatusCollector.FlushStatusUpdate()
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformWakeup()
   at Altiris.Common.Threading.CommonThreadRunner.PerformWakeup()
   at Altiris.NS.Threading.NsThreadRunner.PerformWakeup()
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformLoop(Thread, int, WaitHandle[], int)
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.RunnerThreadProc()
   at System.Threading.ThreadHelper.ThreadStart()

-----------------------------------------------------------------------------------------------------
Date: 4/5/2025 11:18:57 PM, Tick Count: 12187140 (03:23:07.1400000), Size: 3.62 KB
Process: w3wp (33660), Thread ID: 89, Module: System.Data.dll
Priority: 2, Source: SqlCommand.ValidateCommand

Entry 3:

Unspecified Dataloader SQL Exception encountered for inventory data class: 'Managed Software Delivery Policy Progress Status' (a4ecade8-f998-4524-b400-310f94ab4937)

Database context does not have open connection.
   [DeadDatabaseContextException @ Altiris.Database.dll]
   at Altiris.Database.DatabaseContext<>.ProcessException(Exception)
   at Altiris.Database.RetryActionRequest<>.HandleRetryException(IDatabaseContext, IRetriedDatabaseContext, Exception)
   at Altiris.Database.RetryActionRequest<>.Perform(bool)
   at Altiris.Database.DatabaseContext<>.PerformWithDeadlockRetryHelper(int, int, bool, Getter<IDatabaseContext>, Action<IDatabaseContext>, Action<DeadlockRetryArgs>, bool, string, string)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.CommitData(IDataLoaderConnection)

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.NS.ContextManagement.AdminDatabaseContext.ReportException(string, string, Exception)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.CommitData(IDataLoaderConnection)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.CommitData(ImmutableDataRowCacheTransaction, DataClassRowCollection)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.ServeDataEnd()
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.Load(CodeProfiler)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.Load(DataClass, XmlReader, Guid, ProcessingDirective, ResourceImporter)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryLoader.ServeInventoryDataClassDataNode()
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryLoader.Load(CodeProfiler)
   at Altiris.NS.ResourceManagement.DataClassImporter.LoadInventory(XmlReader, bool, bool, Guid)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.ImportDataClassNode(MessagingResource, XmlTextReader, ResourceImporter, DataClassImporter)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.ImportDataClassData(MessagingResource, XmlTextReader, ResourceImporter)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageImpl(string, bool)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageInternal(string, bool)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(string, bool, IItem, IItemMessaging, MessageHeader)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(int, IEventQueueRow, EventQueueRowContent, MessageHeader)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.TryDispatch(int, IEventQueueRow, EventQueueRowContent)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(IEventQueueRow, out bool, EventQueueDispatcher, EventQueue)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.WorkerDispatchStep(DispatchSlot, IAutoCallContextSnapshot)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatchSlotLoop(DispatchSlot, out string)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatchSlotLoop(IMeasureOperation, DispatchSlot)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchSlotThreadProc(object)
   at System.Threading.ThreadHelper.ThreadStart(object)

-----------------------------------------------------------------------------------------------------
Date: 4/7/2025 11:49:21 AM, Tick Count: 9102515 (02:31:42.5150000), Size: 3.53 KB
Process: AeXSvc (25216), Thread ID: 314, Module: Altiris.Database.dll
Priority: 1, Source: DataLoader

Entry 4:

Failed to load inventory.

Database context does not have open connection.
   [DeadDatabaseContextException @ Altiris.Database.dll]
   at Altiris.Database.DatabaseContext<>.ProcessException(Exception)
   at Altiris.Database.RetryActionRequest<>.HandleRetryException(IDatabaseContext, IRetriedDatabaseContext, Exception)
   at Altiris.Database.RetryActionRequest<>.Perform(bool)
   at Altiris.Database.DatabaseContext<>.PerformWithDeadlockRetryHelper(int, int, bool, Getter<IDatabaseContext>, Action<IDatabaseContext>, Action<DeadlockRetryArgs>, bool, string, string)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.CommitData(IDataLoaderConnection)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.CommitData(ImmutableDataRowCacheTransaction, DataClassRowCollection)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.ServeDataEnd()
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.Load(CodeProfiler)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryDataLoader.Load(DataClass, XmlReader, Guid, ProcessingDirective, ResourceImporter)
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryLoader.ServeInventoryDataClassDataNode()
   at Altiris.NS.ResourceManagement.DataClassImporter+InventoryLoader.Load(CodeProfiler)
   at Altiris.NS.ResourceManagement.DataClassImporter.LoadInventory(XmlReader, bool, bool, Guid)

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.NS.ResourceManagement.DataClassImporter.LoadInventory(XmlReader, bool, bool, Guid)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.ImportDataClassNode(MessagingResource, XmlTextReader, ResourceImporter, DataClassImporter)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.ImportDataClassData(MessagingResource, XmlTextReader, ResourceImporter)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageImpl(string, bool)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageInternal(string, bool)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(string, bool, IItem, IItemMessaging, MessageHeader)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(int, IEventQueueRow, EventQueueRowContent, MessageHeader)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.TryDispatch(int, IEventQueueRow, EventQueueRowContent)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(IEventQueueRow, out bool, EventQueueDispatcher, EventQueue)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.WorkerDispatchStep(DispatchSlot, IAutoCallContextSnapshot)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatchSlotLoop(DispatchSlot, out string)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatchSlotLoop(IMeasureOperation, DispatchSlot)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchSlotThreadProc(object)
   at System.Threading.ThreadHelper.ThreadStart(object)

-----------------------------------------------------------------------------------------------------
Date: 4/7/2025 11:49:21 AM, Tick Count: 9102515 (02:31:42.5150000), 
Process: AeXSvc (25216), Thread ID: 314, Module: Altiris.Database.dll
Priority: 1, Source: DatabaseContext.ProcessException

Entry 5:

DatabaseContext finalizer called, which should not happen.
 This: [3/358/1] {Tran, NestedTran, TranOwner, ReadCommitted, Open} id=784593, t='None' ([249e1267d78746568b40518ac5666982]), s=[#3/358] id=688820, AdminDatabaseContext

Message logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.Database.DatabaseContext<>.LogError(int, string, Exception)
   at Altiris.Database.DispositionOps<>.DoFinalizerDispose(T)
   at Altiris.Database.DatabaseContext<>.Dispose(bool)
   at Altiris.Database.DatabaseContext<>.Finalize()

User [HS\srv-smp85srv], AppDomain [AeXSVC.exe]
-----------------------------------------------------------------------------------------------------
Date: 4/8/2025 5:25:32 PM, Tick Count: 73227078 (20:20:27.0780000), Size: 874 B
Process: AeXSvc (17400), Thread ID: 2, Module: Altiris.Database.dll
Priority: 2, Source: AdminDatabaseContext


Entry 6:

Transaction being rolled back automatically during dispose. To disable this warning rollback your transaction explicitly!
 This: [3/358/1] {Tran, NestedTran, TranOwner, ReadCommitted, Open} id=784593, t='None' ([249e1267d78746568b40518ac5666982]), s=[#3/358] id=688820, AdminDatabaseContext

Message logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.Database.DatabaseContext<>.LogError(int, string, Exception)
   at Altiris.Database.DispositionOps<>.DisposeOwnTransaction(T)
   at Altiris.Database.DatabaseContext<>.EmergencyDispose(bool)
   at Altiris.Database.DispositionOps<>.DoEmergencyDispose(T)
   at Altiris.Database.DispositionOps<>.DoFinalizerDispose(T)
   at Altiris.Database.DatabaseContext<>.Dispose(bool)
   at Altiris.Database.DatabaseContext<>.Finalize()

-----------------------------------------------------------------------------------------------------
Date: 4/8/2025 5:25:32 PM, Tick Count: 73227078 (20:20:27.0780000), Size: 1.10 KB
Process: AeXSvc (17400), Thread ID: 2, Module: Altiris.Database.dll
Priority: 2, Source: AdminDatabaseContext


Entry 7:

Automatic rollback during dispose threw unexpected exception.
 This: [3/358/1] {NestedTran, TranOwner, ReadCommitted, Open} id=784593, t='Rolledback' ([249e1267d78746568b40518ac5666982]), s=[#3/358] id=688820, AdminDatabaseContext

There is already an open DataReader associated with this Command which must be closed first.
   [InvalidOperationException @ System.Data.dll]
   at System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand)
   at System.Data.SqlClient.SqlCommand.ValidateCommand(string, bool)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource<object>, string, bool, int, out bool, bool, bool)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Altiris.Database.DatabaseContext<>.ExecuteNonQuery(string, object[])
   at Altiris.Database.DatabaseContext<>.Rollback(bool)
   at Altiris.Database.DispositionOps<>.DisposeOwnTransaction(T)

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.Database.DatabaseContext<>.LogError(int, string, Exception)
   at Altiris.Database.DispositionOps<>.DisposeOwnTransaction(T)
   at Altiris.Database.DatabaseContext<>.EmergencyDispose(bool)
   at Altiris.Database.DispositionOps<>.DoEmergencyDispose(T)
   at Altiris.Database.DispositionOps<>.DoFinalizerDispose(T)
   at Altiris.Database.DatabaseContext<>.Dispose(bool)
   at Altiris.Database.DatabaseContext<>.Finalize()

-----------------------------------------------------------------------------------------------------
Date: 4/8/2025 5:25:32 PM, Tick Count: 73227078 (20:20:27.0780000), Size: 1.73 KB
Process: AeXSvc (17400), Thread ID: 2, Module: System.Data.dll
Priority: 2, Source: AdminDatabaseContext


Entry 8:

Unspecified Dataloader Exception encountered for event data class: fcf38687-fad1-48af-97dc-914aa7c41237

Database context does not have open connection.
   [DeadDatabaseContextException @ Altiris.Database.dll]
   at Altiris.Database.ValidationOps<>.ThrowIfInvalid(T, bool)
   at Altiris.Database.DatabaseContext<>.GetContextImpl(ContextTransactionMode, Nullable<IsolationLevel>, bool)
   at Altiris.Database.DatabaseContext<>.GetContext(ContextTransactionMode, Nullable<IsolationLevel>, bool)
   at Altiris.NS.ResourceManagement.Database.SqlClientConnectionBase..ctor(string, bool)
   at Altiris.NS.ResourceManagement.DataClassRowCollection+EventLoader.Load(Guid, DataClassRows, bool)

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(int, string, string, Exception, string)
   at Altiris.NS.ResourceManagement.DataClassRowCollection+EventLoader.Load(Guid, DataClassRows, bool)
   at Altiris.NS.Logging.EventLog.LogEventDataInternal(IItem, DataClassRows)
   at Altiris.NS.Logging.EventLog.FlushReportedEvents(ICollection<Guid>, bool)
   at Altiris.NS.Logging.EventLog+FlushThreadRunner.SaveEvents(bool)
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformWakeup()
   at Altiris.Common.Threading.CommonThreadRunner.PerformWakeup()
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.PerformLoop(Thread, int, WaitHandle[], int)
   at Altiris.Diagnostics.Threading.DiagnosticsThreadRunner.RunnerThreadProc()
   at System.Threading.ThreadHelper.ThreadStart()

-----------------------------------------------------------------------------------------------------
Date: 4/9/2025 6:34:07 AM, Tick Count: 28034062 (07:47:14.0620000), Size: 1.77 KB
Process: AeXSvc (6416), Thread ID: 31, Module: Altiris.Database.dll
Priority: 1, Source: DataClassRowCollection

Environment

ITMS 8.7.2, 8.7.3

Cause

Known issue. Due to earlier connection errors or code handling limitations, the "DatabaseContext" is being lost and while it's connected it can't be used.  The lost SQL connection(s) were open and holding transaction(s).

At least with this particular scenario these "lost connections" are the ones holding the "locks" and some other queries are then executing for long periods of time, which is effectively slowing down NSE processing and other DB activities.

Resolution

 A fix for this is currently available with our ITMS 8.8 Release.

A pointfix is current available for ITMS 8.7.3 in the following KB article: CUMULATIVE POST ITMS 8.7.3 POINT FIXES  (starting with "SMA_SMP_8_7_3_PF_v4" and later)

Powered by Wolken Software