Not seeing traces from a transaction trace session.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

  Not seeing traces from a transaction trace session over Webview and Workstation, Webview logs shows as below:

 

[ERROR] [WebView] Failed to get TransactionTraceData: com.wily.introscope.spec.server.QueryException: Could not query: Outgoing mailbox is closed. Message cannot be sent
[ERROR] [WebView] Failed to getCorrelatedEvents: com.wily.introscope.spec.server.QueryException: Could not query: Outgoing mailbox is closed. Message cannot be sent
[ERROR] [WebView] Identifier: 1006.1500407684 - receiveMessage() failed for : com.ca.apm.introscope.workstation.webapp.server.models.TransactionTraceModel::2205
java.lang.NullPointerException
    at com.ca.apm.introscope.workstation.webapp.server.models.TTModelHelper.setTTDTimes(TTModelHelper.java:786)
    at com.ca.apm.introscope.workstation.webapp.server.models.TTModelHelper.doTransactionComponentsTree(TTModelHelper.java:308)
    at com.ca.apm.introscope.workstation.webapp.server.models.TransactionTraceModel.doTransactionComponentsTree(TransactionTraceModel.java:844)
    at com.ca.apm.introscope.workstation.webapp.server.models.TransactionTraceModel.receiveMessage(TransactionTraceModel.java:206)
    at com.ca.apm.introscope.workstation.webapp.server.models.ModelManager$ModelRequestThread$1.run(ModelManager.java:410)
    at java.lang.Thread.run(Thread.java:780)

Environment

APM from 10.0 and later.

Cause

 Usually a corrupted or broken traces data file.

Resolution

There are three options to workaround this problem:

Option 1: Recreate the index.
Backup and remove directory "/APM_HOME/traces/index" and restart the Enterprise Manager to regenerate the index directory.

Option 2: Re-index the currently traces directory by using ReindexBuilder found inside APM_HOME/Tools.
Run <APM_HOME\tools> IndexRebuilder.bat APM_HOME\traces.

Option 3: Delete all traces folder if losing traces data is not a problem.

Stop all Enterprise Managers to perform above activities.

Start again after all of the above activities done.

Additional Information

Do the same activity for all Enterprise Managers over your cluster.