APM 10.7 - Trace Database having large index files

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

In our APM 10.7 live instance some index files for Traces DB are of huge size ~45 GB, we believe these are remains from index clean-up jobs that didn’t finish properly because of system ran out of space. Some files from the index directory:

 

 
The timestamps of above two large files roughly correspond with the times where we ran out of disk space on the EM filesystem. Error regarding out of disk space:

[ERROR] [Lucene Merge Thread #0] [Manager] Uncaught Exception in Enterprise Manager: In thread 'Lucene Merge Thread #0' and the message is: org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device

What is best approach to delete such large index files without having significant downtime? Can we delete them manually? Or rebuild index with IndexRebuilder.sh Utility?

Environment

• APM 10.7.x

 

Cause

There were about 10m documents + 17m deleted documents not yet optimized, there are couple of Fields with huge amount of Terms (unique value, ~20m):

• CorCrossProcessData
• contents
• StartTime
• Starttimeinnanos
• TraceID

Indexing this kind of Fields are very expensive, we have option to turn off 2 of them: 

   introscope.enterprisemanager.transactionevents.storage.index.corrids.enable

   introscope.enterprisemanager.transactionevents.storage.index.contents.enable

The corrids property might be useful during support troubleshooting, you can turn off index on contents to reduce the index size.

Resolution

• Recommendation: reduce the size of the Traces DB index to prevent similar situations affecting the index optimization tasks

Apply below recommendation to all the EMs.

1. Stop EM

2. Delete traces/index folder

3. Add to the <EM-HOME>config/IntroscopeEnterpriseManager.properties file, below hidden property:
introscope.enterprisemanager.transactionevents.storage.index.contents.enable=false

4.Start the EM - a new index folder will be created during EM start

NOTE: You can create the index separately and before you start the EM by using the <EM-HOME>/tools/IndexRebuilder tool, please refer to:

https://knowledge.broadcom.com/external/article/186347/reindexing-traces-in-ca-apm.html