APM Collector out of memory due to a corruption in the "Differential Analysis" database

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We have noticed that some APM EM collectors are running out of memory. 

We took a heapdumps and it seems to indicates (based on the mentioned classes) that the problem is related to the APM Baseline engine:

 

Environment

APM 10.8

Cause

The heap dump shows that the execution queue for the baseline metric calculating thread has grown too big.

The below error message in the EM Collector log indicates a possibly baseline db corruption

1/19/23 03:15:53 AM CET [ERROR] [com.ca.apm.baseline.em.Baseline] 

org.garret.perst.StorageError: Object access violation: java.io.StreamCorruptedException: invalid type code: 50
    at org.garret.perst.impl.StorageImpl.loadStub(StorageImpl.java:2831)
...

Resolution

Recommendations:

Option 1) Start all the Collectors with a fresh new Baselines DB

a) stop collector(s)

b) remove the baseline database, its location is defined in the IntroscopeEnterpriseManager.properties, property introscope.enterprisemanager.baseline.database

c) start the collector

 

Option 2) Disable any custom Differential Controls from Webview > Management

For more information refer to Techdocs APM > Differential Analysis section.

 

Additional Information

APM 10.x - Troubleshooting and Best Practices