Getting error 'Unspecified Dataloader Exception encountered for event data class'. Database context invalidated by inner exception.

Products

• IT Management Suite

Issue/Introduction

The following error seems to appear in a regular basis in the NS logs:

Unspecified Dataloader Exception encountered for event data class: fbd5f5ce-caa6-4f73-b83e-03cbe1145e02

Database context invalidated by inner exception
   [Altiris.Database.InvalidDatabaseContextException @ Altiris.Database]
   at Altiris.Database.DatabaseContext`1.ProcessException(Exception exception)
   at Altiris.NS.ResourceManagement.Database.DataLoaderSqlClientInserter.ExecuteBatch(SqlCommand command, IDatabaseContext context, String batch, Int32 done)
   at Altiris.NS.ResourceManagement.Database.DataLoaderSqlClientInserter.CommitBatchImpl(IDatabaseContext ctx, Boolean inOuterTransaction)
   at Altiris.NS.ResourceManagement.Database.DataLoaderSqlClientInserter.CommitBatch()
   at Altiris.NS.ResourceManagement.Database.SqlClientInserter.CommitBatch()
   at Altiris.NS.ResourceManagement.DataClassRowCollection.InternalTableInserterAttempt(NullableDictionary`2 mapResourceInsertInfo, TargetTable targetTable, DataClass dataClass, DataClassRows dataClassRows, IDataLoaderInserter2 inserter, Boolean dataClassIsSplit, Boolean showMessages, Int32 nAttempt)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.InternalTableInserterImpl(IDataLoaderConnection connection, NullableDictionary`2 mapResourceInsertInfo, TargetTable targetTable, String strInserterTable, DataClass dataClass, DataClassRows dataClassRows)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.InternalTableInserter(IDataLoaderConnection connection, NullableDictionary`2 mapResourceInsertInfo, TargetTable targetTable, String strInserterTable, DataClass dataClass, DataClassRows dataClassRows)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.UpdateEventDataClass(DataClass dataClass, DataClassRows rows, IDataLoaderConnection connection, String spResourceUpdateSummary)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.UpdateDataClassImpl(DataClass dataClass, DataClassRows rows, NullableDictionary`2 mapResourceInsertInfo, Boolean historyEnabled, Dictionary`2 resourceGuidToResourceTypeGuid, IDataLoaderConnection connection)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.<>c__DisplayClass28.<UpdateEventDataClass>b__24(IDatabaseContext ctx)
   at Altiris.Database.DatabaseContext`1.RetryAction(Int32 retries, Boolean transacted, Getter`1 getContext, Action`1 action, Action`1 retry, DeadlockRetryArgs& re, Exception& exception, Boolean inTransaction)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetryHelper(Int32 retries, Boolean inTransaction, Getter`1 getContext, Action`1 action, Action`1 retry, Boolean transacted)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetry(Int32 retries, Boolean startNewTransaction, Nullable`1 isolationLevel, Boolean independentContext, Action`1 action, Action`1 retry)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.UpdateEventDataClass(DataClass dataClass, DataClassRows rows, NullableDictionary`2 mapResourceInsertInfo, Boolean historyEnabled, Dictionary`2 resourceGuidToResourceTypeGuid, IDataLoaderConnection connection)
   at Altiris.NS.ResourceManagement.DataClassRowCollection.LoadEvents(Guid classGuid, DataClassRows dataClassRows, Boolean rethrowException)

The maximum system-generated unique value for a duplicate group was exceeded for index with partition ID 72057596276572160. Dropping and re-creating the index may resolve this; otherwise, use another clustering key.
   [System.Data.SqlClient.SqlException @ .Net SqlClient Data Provider]
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Altiris.NS.ResourceManagement.Database.DataLoaderSqlClientInserter.ExecuteBatch(SqlCommand command, IDatabaseContext context, String batch, Int32 done)

SQL Exception details: code=666, line=1

Exception logged from:
   at Altiris.NS.ResourceManagement.DataClassRowCollection.LoadEvents(System.Guid, Altiris.NS.ResourceManagement.DataClassRows, Boolean)
   at Altiris.NS.Replication.ReplicationJobHelper.CreateReplicationJobEvents(String, Altiris.NS.Replication.ReplicationJob)
   at Altiris.NS.Replication.ReplicationJobHelper.AddReplicationJob(String, Altiris.NS.Replication.ReplicationJob, Altiris.NS.Replication.ReplicationJobState)
   at Altiris.NS.Replication.ReplicationManagerService.AddJob(String, Altiris.NS.Replication.ReplicationJob, Altiris.NS.Replication.ReplicationJobState)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr, Object[], Object, Object[]&)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage)
   at System.Runtime.Remoting.Messaging.ServerObjectTerminatorSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage)
   at System.Runtime.Remoting.Messaging.ServerContextTerminatorSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage)
   at System.Runtime.Remoting.Channels.CrossContextChannel.SyncProcessMessageCallback(Object[])
   at System.Runtime.Remoting.Channels.ChannelServices.DispatchMessage(System.Runtime.Remoting.Channels.IServerChannelSinkStack, System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage&)
   at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage(System.Runtime.Remoting.Channels.IServerChannelSinkStack, System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Channels.ITransportHeaders, System.IO.Stream, System.Runtime.Remoting.Messaging.IMessage&, System.Runtime.Remoting.Channels.ITransportHeaders&, System.IO.Stream&)
   at System.Runtime.Remoting.Channels.SoapServerFormatterSink.ProcessMessage(System.Runtime.Remoting.Channels.IServerChannelSinkStack, System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Channels.ITransportHeaders, System.IO.Stream, System.Runtime.Remoting.Messaging.IMessage&, System.Runtime.Remoting.Channels.ITransportHeaders&, System.IO.Stream&)
   at System.Runtime.Remoting.MetadataServices.SdlChannelSink.ProcessMessage(System.Runtime.Remoting.Channels.IServerChannelSinkStack, System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Channels.ITransportHeaders, System.IO.Stream, System.Runtime.Remoting.Messaging.IMessage&, System.Runtime.Remoting.Channels.ITransportHeaders&, System.IO.Stream&)
   at System.Runtime.Remoting.Channels.Tcp.TcpServerTransportSink.ServiceRequest(Object)
   at System.Runtime.Remoting.Channels.SocketHandler.ProcessRequestNow()
   at System.Runtime.Remoting.Channels.SocketHandler.BeginReadMessageCallback(IAsyncResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, Object)
   at System.Net.ContextAwareResult.Complete(IntPtr)
   at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

User [MS\Stg_altns1], AppDomain [AeXSVC.exe]

-----------------------------------------------------------------------------------------------------
Date: 7/24/2017 2:42:55 PM, Tick Count: 15456173 (04:17:36.1730000), Size: 8.14 KB
Process: AeXSvc (564), Thread ID: 35, Module: Altiris.NS.dll
Priority: 1, Source: DataClassRowCollection

Environment

ITMS 8.0 and later

Cause

This is an issue with SQL Server. The following part of the error message in the NS log trace portion leads to the cause of this issue:

The maximum system-generated unique value for a duplicate group was exceeded for index with partition ID 72057596276572160. Dropping and re-creating the index may resolve this; otherwise, use another clustering key.

SQL Exception details: code=666, line=1

Resolution

In order to narrow down what table is having the issue:

1. ​Find the Partition ID reference in the NS log error entry. In this particular example, the Partition ID is 72057596276572160:
   The maximum system-generated unique value for a duplicate group was exceeded for index with partition ID 72057596276572160. Dropping and re-creating the index may resolve this; otherwise, use another clustering key.
2. Run this query using that Partition ID reference:

select schema_name(o.schema_id) as SchemaName, o.name as ObjectName, i.name as IndexName
from sys.partitions p
inner join sys.indexes i on p.object_id = i.object_id
inner join sys.objects o on p.object_id = o.object_id
where p.partition_id = 72057596276572160 and i.index_id = 1;

This should tell us what index is having the issue. In this example the affected index is "idx_Evt_NS_Replication_Rule_ObjectsResourceGuid" for the "Evt_NS_Replication_Rule_Objects" table.

Searching the web for:

message 1:
The maximum system-generated unique value for a duplicate group was exceeded for index with partition ID 72057596276572160. Dropping and re-creating the index may resolve this; otherwise, use another clustering key.

AND

message 2:
SQL Exception details: code=666, line=1

The general recommendation is to:

1. Drop and Recreate the index. 
   One way to do this is:
   • Go to the table in SQL Management Studio,
   • Find the table (in this example "Evt_NS_Replication_Rule_Objects")
   • Expand "Indexes",
   • Select idx_Evt_NS_Replication_Rule_ObjectsResourceGuid
   • and right-click>Script index as>"Drop and Create To">New Query Editor Window.
   • Then just run the query that is generated. 
2. Check as well this article from Microsoft: KB982441 - FIX: You receive error 666 when you rebuild an index online on a partition scheme in SQL Server 2008 or SQL Server 2008 R2