APM 10.7 - Trace Database having large index files
search cancel

APM 10.7 - Trace Database having large index files


Article ID: 203738


Updated On:


CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management


In our APM 10.7 live instance some index files for Traces DB are of huge size ~ 45 GB, we believe these are remains from index clean-up jobs that didn’t finish properly because of system ran out of space. Some files from the index directory : 
-rw-r--r-- 1 root root  46G Aug  5 02:20 _100fe.cfs
-rw-r--r-- 1 root root 2.0M Aug 12 02:00 _100fe_7.del
-rw-r--r-- 1 root root  43G Aug 10 02:13 _1d6a5.cfs
-rw-r--r-- 1 root root 928K Aug 12 02:00 _1d6a5_1.del
-rw-r--r-- 1 root root  15G Aug 12 02:05 _1ij8g.cfs
-rw-r--r-- 1 root root 1.8G Aug 12 08:15 _1jdf8.cfs
-rw-r--r-- 1 root root 173M Aug 12 08:40 _1jfhe.cfs
-rw-r--r-- 1 root root  21M Aug 12 08:43 _1jfom.cfs
-rw-r--r-- 1 root root  19M Aug 12 08:45 _1jfvv.cfs
The timestamps of above two large files roughly correspond with the times where we ran out of disk space on the EM filesystem. Error regarding out of disk space : 
[ERROR] [Lucene Merge Thread #0] [Manager] Uncaught Exception in Enterprise Manager: In thread 'Lucene Merge Thread #0' and the message is: org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
What is best approach to delete such large index files without having significant downtime? Can we delete them manually? Or rebuild index with IndexRebuilder.sh Utility?


APM 10.7.x



Analysis from Development for this specific use case: 

"There were about 10m documents + 17m deleted documents not yet optimized, there are couple of Fields with huge amount of Terms (unique value, ~20m):


Indexing this kind of Fields are very expensive, we have option to turn off 2 of them: 



The corrids property might be useful during support troubleshooting, you can turn off index on contents to reduce the index size "


Recommendation: reduce the size of the Traces DB index to prevent similar situations affecting the index optimization tasks


Apply below recommendation to all the EMs.

1. Stop EM

2. Delete traces/index folder

3. Add to the <EM-HOME>config/IntroscopeEnterpriseManager.properties file, below hidden property:

4.Start the EM - a new index folder will be created during EM start

NOTE: You can create the index separately and before you start the EM by using the <EM-HOME>/tools/IndexRebuilder tool,  see https://knowledge.broadcom.com/external/article/186347/reindexing-traces-in-ca-apm.html


Additional Information


APM Introscope Enterprise Manager - Troubleshooting and Best Practices