Historical trace data is not visible for as long as expected

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

When viewing historical trace data, you will not see data for as long as the configured amount of days in IntroscopeEnterpriseManager.properties, according to this property:

introscope.enterprisemanager.transactionevents.storage.max.data.age=14

This would affect all types of event - traces, errors

Data would be missing in both the Traces and/or Errors tab and in the Historical Query Viewer.

In a cluster environment, the situation would be the same if logging the Workstation / WebView into the MOM or directly to the collector.

There are no obvious error messages - trace data would be retrieved for the last 24 or 48 hours but definitely not for the last 14 days.

If the above property is configured to a value of 14, you will not see 14 individual traces_YYYYMMDD.db files in the /traces folder of each Enteprise Manager

Environment

All APM releases since APM 10x where clamps are configured in apm-events-thresholds-config.xml, any OS, standalone or cluster.

 

Cause

The apm-events-thresholds-config.xml in the /config folder of each Enterprise Manager has a clamp on the amount of disk space used by trace events.

This is the relevant property

introscope.enterprisemanager.transactionevents.storage.max.disk.usage.

By default it is configured to a value of 1024, the value is in MB so this equates to 1GB

Where the individual trace_YYYYMMDD.db files combined file size is greater than 1GB, files will be removed - oldest first, until we reach the storage limit, so the data will not be available to be queried from Workstation/WebView.

Resolution

1. Access the file system of each collector in the cluster and go to the /traces folder.

2. Check each traces_YYYYMMDD and find the file with largest filesize.

3. Based on configuration of:

introscope.enterprisemanager.transactionevents.storage.max.data.age=14

Calculate the required disk usage, and therefore an estimated value for the clamp property.

So, if the largest file is 700 MB, we would do:

700 x 14 = 9800 MB (9.8 GB)

4. Go to apm-events-thresholds.config.xml on each Enterprise Manager and update the value of
introscope.enterprisemanager.transactionevents.storage.max.disk.usage to the calculated value, in this case 9800

<clamp id="introscope.enterprisemanager.transactionevents.storage.max.disk.usage">
            <threshold value="9800"/>
        </clamp>