Historical trace data is not visible for as long as expected

book

Article ID: 40283

calendar_today

Updated On:

Products

APP PERF MANAGEMENT CA Application Performance Management Agent (APM / Wily / Introscope) CUSTOMER EXPERIENCE MANAGER INTROSCOPE

Issue/Introduction

Symptoms:

When viewing historical trace data, you will not see data for as long as the configured amount of days in IntroscopeEnterpriseManager.properties, according to this property:



introscope.enterprisemanager.transactionevents.storage.max.data.age=14



This would affect all types of event - traces, errors

Data would be missing in both the Traces and/or Errors tab and in the Historical Query Viewer.

In a cluster environment, the situation would be the same if logging the Workstation / WebView into the MOM or directly to the collector.

There are no obvious error messages - trace data would be retrieved for the last 24 or 48 hours but definitely not for the last 14 days.

If the above property is configured to a value of 14, you will not see 14 individual traces_YYYYMMDD.db files in the /traces folder of each Enteprise Manager



Environment: 

All APM releases since APM 9.1 where clamps are configured in apm-events-thresholds-config.xml, any OS, standalone or cluster.



Cause:

The apm-events-thresholds-config.xml in the /config folder of each Enterprise Manager has a clamp on the amount of disk space used by trace events.

This is the relevant property

introscope.enterprisemanager.transactionevents.storage.max.disk.usage.

By default it is configured to a value of 1024, the value is in MB so this equates to 1GB

Where the individual trace_YYYYMMDD.db files combined file size is greater than 1GB, files will be removed - oldest first, until we reach the storage limit, so the data will not be available to be queried from Workstation/WebView.



Resolution

1. Access the file system of each collector in the cluster and go to the /traces folder.

2. Check each traces_YYYYMMDD and find the file with largest filesize.

3. Based on configuration of:


introscope.enterprisemanager.transactionevents.storage.max.data.age=14




Calculate the required disk usage, and therefore an estimated value for the clamp property.

So, if the largest file is 700 MB, we would do:



700 x 14 = 9800 MB (9.8 GB)



4. Go to apm-events-thresholds.config.xml on each Enterprise Manager and update the value of
introscope.enterprisemanager.transactionevents.storage.max.disk.usage to the calculated value, in this case 9800


<clamp id="introscope.enterprisemanager.transactionevents.storage.max.disk.usage">
            <threshold value="9800"/>
        </clamp>




Additional Information:

https://docops.ca.com/ca-apm/10-2/en/installing/apm-sizing-and-performance-guide/ca-apm-data-storage-requirements

See: traces.db Disk Space Calculation Example for details of how much disk space traces could use and how high the clamp could need to be configured at to avoid data loss.


Environment

Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main
Component: