How can we resolve slow Experience View and slow APM Team Center in general?

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

We have noticed the following symptoms in Experience View and APM Team Center in general:

- ATC experience view is very slow (minute to render).

- ATC notebook is very slow (minute to render).

- ATC historical view in map is very slow to load (minutes to load).

- ATC live mode map is very slow to load (minutes to load).

Environment

APM 10.5.x

Cause

  - Old release is deployed.

  - Database is undersized and is lacking cache.

  - EM memory is undersized (GC overhead).

Resolution

To resolve this problem, please install latest Hotfix for your release since a lot of performance improvements were done for 10.5.1 and 10.5.2.

In particular, one should look for 10.5.1HF44 hotfix and 10.5.2SP2HF1 hotfix as a minimum. If upgrade will not help completely, then refer to the Technical Document "How can we optimize the database when the database size is growing very quickly?";. See "Additional Information" section below to access the Technical Document.

 

Additional checks

- Check Query Size:
introscope.enterprisemanager.query.datapointlimit=100000
introscope.enterprisemanager.query.returneddatapointlimit=100000

- Check Message Queue Size and eventually increase:
transport.outgoingMessageQueueSize=8000
transport.override.isengard.high.concurrency.pool.min.size=10
transport.override.isengard.high.concurrency.pool.max.size=10

- Check that SmartStor database is indeed on dedicated disk controller, as following property value is seen to be defined in all MOM and Collectors:
introscope.enterprisemanager.smartstor.dedicatedcontroller=true

- Check that Smartstor tier3 is set to the period you need, and not longer:
introscope.enterprisemanager.smartstor.tier3.frequency
introscope.enterprisemanager.smartstor.tier3.age

- Check Memory:
Make the Xms setting the same as Xmx for the MOM.

- Update Garbage Collection:
Investigate changing from concurrent mark sweep to garbage first, G1C, which is more efficient for larger heaps. Please be advised that changing GC is very fundamental and can have adverse effects. Hence, be thorough in understanding and adhering to Java documentation as well as in testing!

Replace these 2 parameters
XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:CMSInitiatingOccupancyFraction=50

With just this 1 parameter:
XX:+UseG1GC

With G1C you are also able to increase maximum heap size with less GC pause (or “freeze” impact) as per Oracle documentation.

- Check Traces:
Reduce number of persisted traces, consider clamping number of Transaction Traces sent by agents to EM Collectors. In the EM property file “apm-events-thresholds-config.xml” edit the clamp:
introscope.enterprisemanager.agent.trace.limit: e.g. reduce from 1000 to 100 or 50

Number of traces triggered by differential analysis. Increase auto tracing triggering threshold from 30 to 15 to reduce auto generated traces from DA:
introscope.enterprisemanager.baseline.tracetrigger.variance.intensity.trigger=15.

- Check for Excessive Meta Data:
Check for excessive meta data in the supportability “Enterprise Manager” type view. The ratio between live metrics and historical metrics should not be larger than 2 to 2.5
If it is larger, you should either consider pruning/cleaning it (using SmartStor Tools), reducing it (setting a shorter period for tier 3 data retention for at least 24 hours for aging out to kick in),
or upgrading to 10.5.2 (in which meta data is implemented and processed with Rocks DB which is generally vastly more efficient as well almost insensitive to excessive meta data).

 

Additional Information

- How can we optimize the database when the database size is growing very quickly?
https://support.ca.com/us/knowledge-base-articles.TEC1639888.html

- How can we run the Hotfix APM10.5.2HF17 script?
https://support.ca.com/us/knowledge-base-articles.TEC1772714.html

- How can we run the Hotfix APM10.5.1HF44 script?
https://support.ca.com/us/knowledge-base-articles.TEC1191544.html

- How to purge or reduce size of APM database in Postgres?
https://support.ca.com/us/knowledge-base-articles.TEC596922.html

- How do I cleanup SmartStor data?
https://support.ca.com/us/knowledge-base-articles.TEC1337681.html