High memory usage, on AM Manager, crashing or DSM Explorer erroring with AMM003004
Article ID: 72635
Updated On:
Products
CA Client Automation - IT Client Manager
CA Client Automation
Issue/Introduction
AMManager (amObjectManager.exe) process is consuming high amounts of memory or crashing on a regular basis. DSM Explorer users may be experiencing "Message timeout [AMM003004]" error.
Environment
Client Automation - Any version.
Cause
AM Object Manager (ammanager) is the API into the database for DSM Explorer to request asset data from the database. AMmanager is responsible for retrieving computer data, group data and performing query evaluations to return requested data to the user portal (DSM Explorer or Web Console). As a 32-bit process, amobjectmanager.exe is restricted by a 2GB memory ceiling. Due to heavy volumes of requests, or a DSM Query that is corrupted, complex or returns excessive amounts of data, ammanager may experience high memory usage or crash. Each instance of the problem may have a unique cause or contributing factors. Common causes are invalid, corrupted or complex DSM queries, or queries that return too much data. Other causes include too many registered computers, user accounts or software delivery user profiles, or too many dynamic groups that may be evaluating too frequently.
Resolution
There are several common causes and contributing factors that need to be checked...
Common Cause #1: DSM Queries
- Review the recently created DSM queries by running the following query in SQL:
select * from ca_query_def order by creation_date desc
If you suspect or would like to eliminate a recently created DSM query, you can run the following statements to remove the query from the database:
delete from ca_query_def_contents where query_uuid in (select query_uuid from ca_query_def where query_uuid=<UUID of query to remove>)
delete from ca_query_def where query_uuid=<UUID of query to remove>
- Review all the complex DSM queries.
Complex DSM Queries are queries created in ITCM that have several arguments and require additional storage in the database. Simple queries are stored in the ca_query_def table. Queries that have a large number of parameters or are exceeding complex are spilled over into the ca_query_def_contents table in the database.
To review complex DSM queries, run the following select statement:
select * from ca_query_def where query_uuid in (select query_uuid from ca_query_def_contents)
Follow the same method for removing any query that you suspect may be corrupted or causing the issue:
delete from ca_query_def_contents where query_uuid in (select query_uuid from ca_query_def where query_uuid=<UUID of query to remove>)
delete from ca_query_def where query_uuid=<UUID of query to remove>
- Upon startup, ammanager will perform a cache of the DSM query objects in the database. Therefore, the more queries that are built-up and stored over time, may be contributing to the issue. It is highly recommended to periodically review all the DSM queries and eliminate anything that is obsolete or not used. The less DSM queries there are, the less there is for ammanager to cache upon startup, and the less overall memory used.
Common Cause #2: Dynamic Group Evaluations
- Review of all the dynamic computer groups.
Dynamic computer groups are the groups that are query based, and are evaluated periodically to add or remove computers as members. Over time, the number of these dynamic groups can begin to pile up, which will contribute to the overall volume of requests sent to ammanager to perform these evaluations in the database.
Review all the dynamic groups in DSM Explorer, and eliminate the ones that are obsolete or no longer used. For each dynamic group removed, consider also deleting the associated DSM query from the All Queries list. Every group and query removed will be less information for ammanager needs to cache, and less queries to evaluate in the database, thereby increasing performance of ITCM.
For dynamic groups that you are keeping, review the properties of the groups evaluation. Having too many dynamic groups that are evaluating on "All Engines" will adversely affect ammanager and the overall performance of DSM Explorer. The best practice is to NEVER have any groups evaluate on "All Engines", as this will significantly reduce performance, as the group will constantly be evaluated in the background.
To check for dynamic groups set to evaluate on ALL ENGINES, run the following query in the database:
select label, creation_user from ca_group_def where query_uuid is not null and evaluation_uuid is null order by label
It is HIGHLY recommended to designate a specific evaluation engine for each of these queries.
Similarly, a review of the evaluation frequency of dynamic groups needs to be checked. Note that DSM Explorer displays evaluation frequency time in minutes, while the database stores this value in number of seconds.
Run the following query to list dynamic groups by evaluation frequency:
select label, eval_freq, creation_user from ca_group_def where query_uuid is not null and evaluation_uuid is not null order by label
When reviewing the results, consider how often you really need the members of the particular group re-evaluated. In most cases, once per day is usually fine. Some cases may call for every hour. Also take into consideration that groups can be evaluated on-demand in DSM Explorer, outside the normal evaluation period.
Common Cause #3: Obsolete Registered Computers, User Accounts or Software Delivery Profiles
Check the count overall counts of registered objects in the database. Once again, without regular maintenance to purge obsolete computers, users or software delivery profiles, these objects can pile up in the database, and cause ammanager to consume memory needlessly to cache them...
-- Computers
select count(*) from ca_agent where agent_type=1
Last Updated >90 days: select count(*) from ca_agent where agent_type=1 and last_run_date<DATEDIFF(s,'19700101',GETUTCDATE())-7776000
-- Discovered Users
select count(*) from ca_agent where agent_type=2
Last Updated >90 days: select count(*) from ca_agent where agent_type=2 and last_run_date<DATEDIFF(s,'19700101',GETUTCDATE())-7776000
-- Software Delivery user profiles:
select count(*) from ca_agent where agent_type=4
Last Updated >90 days: select count(*) from ca_agent where agent_type=4 and last_run_date<DATEDIFF(s,'19700101',GETUTCDATE())-7776000
Note: For registered computers and discovered users, you can set a policy to automatically delete objects that haven't reported in a specified period of time. DSM Explorer --> Queries --> Wizards --> Obsolete Assets.
Common Cause #1: DSM Queries
- Review the recently created DSM queries by running the following query in SQL:
select * from ca_query_def order by creation_date desc
If you suspect or would like to eliminate a recently created DSM query, you can run the following statements to remove the query from the database:
delete from ca_query_def_contents where query_uuid in (select query_uuid from ca_query_def where query_uuid=<UUID of query to remove>)
delete from ca_query_def where query_uuid=<UUID of query to remove>
- Review all the complex DSM queries.
Complex DSM Queries are queries created in ITCM that have several arguments and require additional storage in the database. Simple queries are stored in the ca_query_def table. Queries that have a large number of parameters or are exceeding complex are spilled over into the ca_query_def_contents table in the database.
To review complex DSM queries, run the following select statement:
select * from ca_query_def where query_uuid in (select query_uuid from ca_query_def_contents)
Follow the same method for removing any query that you suspect may be corrupted or causing the issue:
delete from ca_query_def_contents where query_uuid in (select query_uuid from ca_query_def where query_uuid=<UUID of query to remove>)
delete from ca_query_def where query_uuid=<UUID of query to remove>
- Upon startup, ammanager will perform a cache of the DSM query objects in the database. Therefore, the more queries that are built-up and stored over time, may be contributing to the issue. It is highly recommended to periodically review all the DSM queries and eliminate anything that is obsolete or not used. The less DSM queries there are, the less there is for ammanager to cache upon startup, and the less overall memory used.
Common Cause #2: Dynamic Group Evaluations
- Review of all the dynamic computer groups.
Dynamic computer groups are the groups that are query based, and are evaluated periodically to add or remove computers as members. Over time, the number of these dynamic groups can begin to pile up, which will contribute to the overall volume of requests sent to ammanager to perform these evaluations in the database.
Review all the dynamic groups in DSM Explorer, and eliminate the ones that are obsolete or no longer used. For each dynamic group removed, consider also deleting the associated DSM query from the All Queries list. Every group and query removed will be less information for ammanager needs to cache, and less queries to evaluate in the database, thereby increasing performance of ITCM.
For dynamic groups that you are keeping, review the properties of the groups evaluation. Having too many dynamic groups that are evaluating on "All Engines" will adversely affect ammanager and the overall performance of DSM Explorer. The best practice is to NEVER have any groups evaluate on "All Engines", as this will significantly reduce performance, as the group will constantly be evaluated in the background.
To check for dynamic groups set to evaluate on ALL ENGINES, run the following query in the database:
select label, creation_user from ca_group_def where query_uuid is not null and evaluation_uuid is null order by label
It is HIGHLY recommended to designate a specific evaluation engine for each of these queries.
Similarly, a review of the evaluation frequency of dynamic groups needs to be checked. Note that DSM Explorer displays evaluation frequency time in minutes, while the database stores this value in number of seconds.
Run the following query to list dynamic groups by evaluation frequency:
select label, eval_freq, creation_user from ca_group_def where query_uuid is not null and evaluation_uuid is not null order by label
When reviewing the results, consider how often you really need the members of the particular group re-evaluated. In most cases, once per day is usually fine. Some cases may call for every hour. Also take into consideration that groups can be evaluated on-demand in DSM Explorer, outside the normal evaluation period.
Common Cause #3: Obsolete Registered Computers, User Accounts or Software Delivery Profiles
Check the count overall counts of registered objects in the database. Once again, without regular maintenance to purge obsolete computers, users or software delivery profiles, these objects can pile up in the database, and cause ammanager to consume memory needlessly to cache them...
-- Computers
select count(*) from ca_agent where agent_type=1
Last Updated >90 days: select count(*) from ca_agent where agent_type=1 and last_run_date<DATEDIFF(s,'19700101',GETUTCDATE())-7776000
-- Discovered Users
select count(*) from ca_agent where agent_type=2
Last Updated >90 days: select count(*) from ca_agent where agent_type=2 and last_run_date<DATEDIFF(s,'19700101',GETUTCDATE())-7776000
-- Software Delivery user profiles:
select count(*) from ca_agent where agent_type=4
Last Updated >90 days: select count(*) from ca_agent where agent_type=4 and last_run_date<DATEDIFF(s,'19700101',GETUTCDATE())-7776000
Note: For registered computers and discovered users, you can set a policy to automatically delete objects that haven't reported in a specified period of time. DSM Explorer --> Queries --> Wizards --> Obsolete Assets.
Feedback
Yes
No
Powered by
