APM 10.7 - Trace Database having large index files

Products

CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management

Issue/Introduction

In our APM 10.7 live instance some index files for Traces DB are of huge size ~ 45 GB, we believe these are remains from index clean-up jobs that didn’t finish properly because of system ran out of space. Some files from the index directory :

-rw-r--r-- 1 root root 46G Aug 5 02:20 _100fe.cfs
-rw-r--r-- 1 root root 2.0M Aug 12 02:00 _100fe_7.del
-rw-r--r-- 1 root root 43G Aug 10 02:13 _1d6a5.cfs
-rw-r--r-- 1 root root 928K Aug 12 02:00 _1d6a5_1.del
-rw-r--r-- 1 root root 15G Aug 12 02:05 _1ij8g.cfs
-rw-r--r-- 1 root root 1.8G Aug 12 08:15 _1jdf8.cfs
-rw-r--r-- 1 root root 173M Aug 12 08:40 _1jfhe.cfs
-rw-r--r-- 1 root root 21M Aug 12 08:43 _1jfom.cfs
-rw-r--r-- 1 root root 19M Aug 12 08:45 _1jfvv.cfs

The timestamps of above two large files roughly correspond with the times where we ran out of disk space on the EM filesystem. Error regarding out of disk space :

[ERROR] [Lucene Merge Thread #0] [Manager] Uncaught Exception in Enterprise Manager: In thread 'Lucene Merge Thread #0' and the message is: org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device

What is best approach to delete such large index files without having significant downtime? Can we delete them manually? Or rebuild index with IndexRebuilder.sh Utility?

Environment

APM 10.7.x

Cause

Analysis from Development for this specific use case:

"There were about 10m documents + 17m deleted documents not yet optimized, there are couple of Fields with huge amount of Terms (unique value, ~20m):

CorCrossProcessData
contents
StartTime
Starttimeinnanos
TraceID

Indexing this kind of Fields are very expensive, we have option to turn off 2 of them:

introscope.enterprisemanager.transactionevents.storage.index.corrids.enable

introscope.enterprisemanager.transactionevents.storage.index.contents.enable

The corrids property might be useful during support troubleshooting, you can turn off index on contents to reduce the index size "

Resolution

Recommendation: reduce the size of the Traces DB index to prevent similar situations affecting the index optimization tasks

Apply below recommendation to all the EMs.

1. Stop EM

2. Delete traces/index folder

3. Add to the <EM-HOME>config/IntroscopeEnterpriseManager.properties file, below hidden property:
introscope.enterprisemanager.transactionevents.storage.index.contents.enable=false

4.Start the EM - a new index folder will be created during EM start

NOTE: You can create the index separately and before you start the EM by using the <EM-HOME>/tools/IndexRebuilder tool, see https://knowledge.broadcom.com/external/article/186347/reindexing-traces-in-ca-apm.html

Additional Information

APM Introscope Enterprise Manager - Troubleshooting and Best Practices
https://knowledge.broadcom.com/external/article/93176/introscope-enterprise-manager-troublesho.html