SMP server is maxing out the memory usage for aexsvc and w3wp services

Products

IT Management Suite

Issue/Introduction

This customer started noticing in the last few days that his SMP Server is consuming more memory than usual. He doesn't have that many client machines (between client machines and servers he has around 1000 systems).
aexsvc.exe (Altiris Service) goes from ~40mb of ram up to ~13gb.

If he reboots the SMP server, the issue stops but just for a day or so and it comes back. Then he can't move around in the SMP console.

Environment

ITMS 8.5, 8.6

Cause

Known issue.

Case 1:

We found out that making our services run in "server garbage collect mode" (which is recommended, actually), can make them consume too much memory.

Case 2:

Usage of "Item Trackers". Previous Admins from this customer were probably trying to capture only changes done by SMP Console - this is why their tracking is set to the IIS process.
As we found out after reviewing the aexsvc.exe memory dump, there seem to be too many items that are being saved during the "GetClientPolicies" (going through IIS) request from the client. Archiving (without deletion) for this item tracking events is the issue, because .NET API to manipulate archives eats too much memory. This is what would be refactored in next 8.6 RU3 release.

Resolution

We had added some improvements with our ITMS 8.6 RU3 release to alleviate some of the common issues addressed by the following suggestions:

Suggestion 1:

1. Locate the AeXSvc.exe.config file under ...\program files\Altiris\Notification Server\Bin

2. Edit it manually, replacing the line:

with

3. Restart Altiris Service

Note: any other process/service of ITMS, which consumes too much memory - can also be instrumented like this.

Suggestion 2:

Check the "Item Tracking" settings on the SMP Console (under Settings>Notification Server>Internals>Item Tracking>Item Trackers) for the affected SMP Server. See if there are any "Item Trackers" enabled. By default, we don't provide or enable item trackers. This feature was introduced with ITMS 8.5 release and customers can add them if the choose to. The idea was to capture moments, when an item being changed by some rules and save info about it somewhere. See "Auditing through Item Trackers Symantec Management Platform 8.5" (KB 173483)
If "Item Trackers" are in use, make sure "archive" and "purge" of the data is more aggressive and tracking items scope to be more conservative. These settings can be found at Settings>Notification Server>Internals>Item Tracking>Global Item Tracking Settings. By default, the captured data (as files) is set like this:

If Item Trackers are in use and there is a high memory usage, it is most likely the customer created "item trackers" to capture too many events, which flooded the file system and the archiving schedule of it just consumed too much memory.

If "Item Trackers" are in use, for now:

* delete these collected files (at the location of tracking, which by default is "C:\ProgramData\Symantec\SMP\Audit")

* turn OFF tracking entirely or make it capture much less data and purge data older than 1 day or something similar.

Note: If the customer is not sure if they still need these item tracking objects anymore - our advice would be to manually delete everything under the "tracking files location" (C:\programdata\symantec\smp\audit folder) and turn it off entirely.

Just turn OFF tracking entirely on "Global Item Tracking Settings" page. This will prevent both collecting the data and performing archiving, which kills the system at that point.

Note: The customer can turn it ON again if they need to after they have upgraded to ITMS 8.6 RU3 (we are planning to add some improvements in our code so we don't consume too much memory).

The schedule to run "Item Tracking Maintenance" is "NS.Daily" under Windows Task Scheduler.
We don't have any special UI to adjust the scheduling for this maintenance, but if you turn OFF tracking, it will not perform.

Improvements done for our 8.6 RU3 next release:

1. Archiving will make "daily" zip files, collecting only changes for each day - not everything going to be in same archive. This will speedup action, as well as minimize memory usage. Also, it is trying to not process files, that already have archives for that day.

2. More options to allow/deny in trackers should help to "filter out" activities, which are not intended to be captured even with excessively wide settings by item class.

3. Option to delete archives after some time is added - not to keep new "daily" archives for too long.