Error: "An unknown exception was thrown on server side. Failed to timely stop the task. Task was detached." when gathering Full Inventory Task is taking too long

Products

• IT Management Suite
• Inventory Solution

Issue/Introduction

You are trying to run a Full Inventory Task as part of your Imaging job - As part of imaging jobs, we like to gather inventory so that it is reported back to the SMP Server even if the computer is powered off and not used for weeks after imaging.

However, it takes too long, and the task times out with the following error:

An unknown exception was thrown on server side.

Failed to timely stop the task. Task was detached.

The following is provided as an example of what the agent logs showed as the Inventory task process:

1. The start time for this "Gather Inventory Full SW Only" is 10:48:41am

Task Execution Engine: Task instance count for parent 'f61c572d-1842-4eb8-b428-01ff923910b5' initialized to 1.
-----------------------------------------------------------------------------------------------------
Date: 2/11/2022 10:48:41 AM, Tick Count: 227828 (00:03:47.8280000), Size: 347 B
Process: AeXNSAgent.exe (6768), Thread ID: 7144, Module: client task agent.dll
Priority: 8, Source: Client Task Agent

 

Core: Task added to the queue: f61c572d-1842-4eb8-b428-01ff923910b5, type: ts
-----------------------------------------------------------------------------------------------------
Date: 2/11/2022 10:48:41 AM, Tick Count: 227828 (00:03:47.8280000), Size: 314 B
Process: AeXNSAgent.exe (6768), Thread ID: 7144, Module: client task agent.dll
Priority: 8, Source: Client Task Agent

 

Task Status Engine: Updating new task instance 'f61c572d-1842-4eb8-b428-01ff923910b5' info, type: ts, name: Gather Inventory Full SW Only, description: Standard System Inventory, hidden: No
-----------------------------------------------------------------------------------------------------
Date: 2/11/2022 10:48:41 AM, Tick Count: 227828 (00:03:47.8280000), Size: 425 B
Process: AeXNSAgent.exe (6768), Thread ID: 7144, Module: client task agent.dll
Priority: 8, Source: Client Task Agent

 

2. From what we could tell, the actual "Gather Inventory Full SW Only" ended at 12:20:51 PM:

Task History Engine: Processing finished task instance 'f61c572d-1842-4eb8-b428-01ff923910b5', type: ts, name: Gather Inventory Full SW Only, description: Standard System Inventory, hidden: No
-----------------------------------------------------------------------------------------------------
Date: 2/11/2022 12:20:51 PM, Tick Count: 3909671 (01:05:09.6710000), Size: 430 B
Process: AeXNSAgent.exe (6768), Thread ID: 7660, Module: client task agent.dll
Priority: 8, Source: Client Task Agent

 

3. One thing that we noticed while troubleshooting this issue, it seems another inventory was running before the one that came with their imaging job:

Task queue woke but job still in progress. Job belongs to task: Gather Inventory Delta package: (null) trigger: c178eecb-ceec-4a32-9864-17f4e75220f9 started: 2022-02-11 11:16:06
-----------------------------------------------------------------------------------------------------
Date: 2/11/2022 10:46:22 AM, Tick Count: 219796 (00:03:39.7960000), Size: 418 B
Process: AeXNSAgent.exe (6768), Thread ID: 6800, Module: smfagent.dll
Priority: 4, Source: CDeliveryPolicy::RunJobQueue()

"Gather Inventory Delta" is not the one that came with the customer job (in this example, it is called "Gather Inventory Full SW Only").

Environment

ITMS 8.6 RU1, 8.6 RU2

Cause

Known issue. There were overlapping Inventories and causing them to queue longer than needed the one that came from the imaging job.

The issue found was:

• Inventory has a built-in "inconsistency" detection mechanism. It detects cases when collected dataclasses for some reason are not processed by the NS server. In this case, a special internal inventory collection job is launched to collect "inconsistent" dataclasses.
• Reimaging may be a trigger that causes inconsistency. The state of collected inventory on the client becomes older than the state saved on the server. So "inconsistency" detection mechanism launches inventory collection on the background.
• To prevent concurrent execution of several inventory jobs there is a mutex. But in the case of "remediation" job this mutex was not always released correctly. So new inventory job that was executed after the "remediation" job may be hung forever trying to obtain mutex lock.

Resolution

This issue was been fixed with the ITMS 8.6 RU3 release.