APM 10.7 - Trace Database having large index files
search cancel

APM 10.7 - Trace Database having large index files


Article ID: 203738


Updated On:


CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management


In our APM 10.7 live instance some index files for Traces DB are of huge size ~45 GB, we believe these are remains from index clean-up jobs that didn’t finish properly because of system ran out of space. Some files from the index directory:

The timestamps of above two large files roughly correspond with the times where we ran out of disk space on the EM filesystem. Error regarding out of disk space:
[ERROR] [Lucene Merge Thread #0] [Manager] Uncaught Exception in Enterprise Manager: In thread 'Lucene Merge Thread #0' and the message is: org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
What is best approach to delete such large index files without having significant downtime? Can we delete them manually? Or rebuild index with IndexRebuilder.sh Utility?


  • APM 10.7.x



There were about 10m documents + 17m deleted documents not yet optimized, there are couple of Fields with huge amount of Terms (unique value, ~20m):

  • CorCrossProcessData
  • contents
  • StartTime
  • Starttimeinnanos
  • TraceID

Indexing this kind of Fields are very expensive, we have option to turn off 2 of them: 



The corrids property might be useful during support troubleshooting, you can turn off index on contents to reduce the index size.


  • Recommendation: reduce the size of the Traces DB index to prevent similar situations affecting the index optimization tasks

Apply below recommendation to all the EMs.

1. Stop EM

2. Delete traces/index folder

3. Add to the <EM-HOME>config/IntroscopeEnterpriseManager.properties file, below hidden property:

4.Start the EM - a new index folder will be created during EM start

NOTE: You can create the index separately and before you start the EM by using the <EM-HOME>/tools/IndexRebuilder tool, please refer to: