High AeXSvc Service CPU usage caused by spGetObsoleteInventoryConsistencyRecords

Products

• IT Management Suite

Issue/Introduction

The customer recently upgraded to ITMS 8.6. After that, the customer has experienced high memory usage, specially by the AeXSvc service. 

Environment

ITMS 8.6

Cause

Known issue. Dump memory analysis showed that this issue was caused by our code while calling SQL procedure "spGetObsoleteInventoryConsistencyRecords". It was producing ~37 million records.

The cleanup code, that last succeeded (by ns.log entries), was executing 2+ days to cleanup this data and yet failed:

"9/5/2022 4:41:05 PM","Unspecified Dataloader SQL Exception encountered for inventory data class: 'Client Data Recency' (b14d4b97-dd8f-4d63-8a89-75e544ac5c8b)
...

The wait operation timed out
...

"9/5/2022 4:41:05 PM","Operation completed: Detecting obsolete inventory consistency records, total time=2.14:30:15.1837968","InventoryInconsistencyDetectionPolicy","AeXSVC.exe","15","Informational"

The table that probably have too many records is "Inv_Client_Data_Recency"

Resolution

This issue has been reported to our Broadcom Development team. A fix for this has been added to our ITMS 8.7 release.

A pointfix for customers with ITMS 8.6 RU3 is available. See KB 249427 "CUMULATIVE POST ITMS 8.6 RU3 POINT FIXES"

The changes added to the improved "spGetObsoleteInventoryConsistencyRecords" should allow only delete of 10,000 records on each attempt of the scheduled run and avoid a time-out. It will take a couple of runs of the cleanup process to address a large set of entries that needs to be cleaned up.

The NS logs will show an entry saying something like:
Operation completed: Deleting obsolete inventory consistency records, deleted=2358709