Reindexing Traces in CA APM

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

When running 10.7 APM, the traces directory got full.
So we have run indexRebuilder.sh to clear old traces.
However, it is running for long time. We are losing data as the EM is stopped.

Can we stop the re indexing, would there be a risk of losing trace data?

Environment

Release : 10.7.0 (valid for any previous release)

Component : APM Enterprise Manager

Cause

The indexRebuilder is reading through all of the stored trace information to create a new index.
There is no data loss in stopping the process but traces will not be found if the index is not created.

The indexRebuilder.sh/bat scripts run with default Java heap allocation by default.
When there are potentially millions of trace records to process, the heap memory of the reindexing process should be increased to speed it up.
Since the Enterprise Manager must be stopped to perform this, you can allocate as much memory to this process as you would normally do for the Enterprise Manager. You can allocate more memory than you would normally give the Enterprise Manager as long as there is enough free memory on the machine.

Note that this process does not remove any traces, but, unlike the traces db files themselves, the trace index disk space is not clamped. so running this process can reduce used disk space.

Resolution

This is a two-step process, unless you already know the Enterprise Manager's heap allocation or maximum available memory on the machine

• Check current EM heap allocation

In terms of Enterprise Manager heap memory configuration, running on Linux, it will be set in Introscope_Enterprise_Manager.lax, here set to 8GB

     lax.nl.java.option.additional=-Xms8192m -Xmx8192m -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseG1GC -          XX:MaxGCPauseMillis=200 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/  -Xss512k -Djava.io.tmpdir=./work


Checking the -Xms and -Xmx values in there to set in the index rebuilder script.

For a Windows service, you would check EMService.conf in the bin folder of the software for these settings. This example again sets the memory at 8GB

     # Initial Java Heap Size (in MB)
     wrapper.java.initmemory=8192

     # Maximum Java Heap Size (in MB)
     wrapper.java.maxmemory=8192

• Increase heap in index rebuild script

If you have a Linux machine, with file IndexRebuilder.sh you will have the line:

          "$JAVA_HOME/bin/java" -Declipse.consoleLog=false -jar "$installDir/launcher.jar" -install "$installDir/product/enterprisemanager" -application com.wily.introscope.eclipseapp.IndexRebuilder "$1"

If you have a Windows machine, with file IndexRebuilder.bat, you will have the line:

         "%JAVA_HOME%\bin\java.exe" -Declipse.consoleLog=false -jar "%INSTALLDIR%\launcher.jar" -install "%INSTALLDIR%\product\enterprisemanager" -application com.wily.introscope.eclipseapp.IndexRebuilder %1


For the index rebuilding to run more quickly, as suggested, increase the heap, allocating usually the same amount of heap you give to the Enterprise Manager, using -Xms and -Xmx settings

As an example, to allocate 8GB heap, you would change the line so it looks like this:

Linux:

          "$JAVA_HOME/bin/java" -Xms8192m -Xmx8192m -Declipse.consoleLog=false -jar "$installDir/launcher.jar" -install "$installDir/product/enterprisemanager" -application com.wily.introscope.eclipseapp.IndexRebuilder "$1"

Windows:

          "%JAVA_HOME%\bin\java.exe" -Xms8192m -Xmx8192m -Declipse.consoleLog=false -jar "%INSTALLDIR%\launcher.jar" -install "%INSTALLDIR%\product\enterprisemanager" -application com.wily.introscope.eclipseapp.IndexRebuilder %1

Additional Information

Traces are stored on each Enterprise Manager, there will be minimal entries on a MOM but will be more on a collector.