Historical trace information not available

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

   Problem:

    Historical transaction trace data is not available either in WebView or Workstation. The header information is available but the details will not be displayed

    and  you will see an error as follows:

  

<Please see attached file for image>

 

   This is generated in one of two ways:

     a) by putting the Investigator into historical mode then selecting a time frame before today and selecting the Traces or Errors tab and choosing a particular event,

     b) or by using the Historical Query Viewer to search for any type of event and selecting a time frame for a day before the current one.

   Symptoms

• Today's traces will display with no problem.
• If you take a copy of the traces folder and review with the Lucene tool, all trace information will be available.
• If you take the trace information to another Enterprise Manager, you may not get the same problem.
• You would also notice this error in the Enterprise Manager log in VERBOSE logging

 

 [VERBOSE] [PO:main Mailman 3] [Manager.TransactionTracer] Failed to get historical trace: 1433938415115:8081
  com.wily.introscope.server.enterprise.entity.transactiontransactiontrace.storage.HistoricalPerstStorage.open(HistoricalPerstStorage.java:213)
  at com.wily.introscope.server.enterprise.entity.transactiontrace.storage.MultiStorePerstStorage.openHistoricalStorageIfNecessary(MultiStorePerstStorage.java:728)
  at com.wily.introscope.server.enterprise.entity.transactiontrace.storage.MultiStorePerstStoragetrace.storage.TransactionTraceStorageException: Unable to open the      Perst backing store: /opt/apm/traces/traces_20150610.db
 at com.wily.introscope.server.enterprise.entity..getTraceFromHistoricalStorage(MultiStorePerstStorage.java:716)
 at com.wily.introscope.server.enterprise.entity.transactiontrace.storage.MultiStorePerstStorage.doGetHistoricalTrace(MultiStorePerstStorage.java:592)
 at com.wily.introscope.server.enterprise.entity.transactiontrace.storage.MultiStorePerstStorage.getHistoricalTrace(MultiStorePerstStorage.java:353)
 at com.wily.introscope.server.enterprise.entity.transactiontrace.storage.TransactionTraceStorageSystem.getHistoricalTrace[..]
 Caused by: org.garret.perst.StorageError: Failed to locate org.garret.perst.impl.AltBtree$BtreePageOfString: java.lang.ClassNotFoundException:  org.garret.perst.impl.AltBtree$BtreePageOfString (com.wily.isengard.tunnelingservice)
 at org.garret.perst.impl.ClassDescriptor.loadClass(ClassDescriptor.java:353)
 at org.garret.perst.impl.ClassDescriptor.onLoad(ClassDescriptor.java:358)
 [..]

  Note the java.lang.ClassNotFoundException for org.garret.perst.impl.AltBtree$BtreePageOfString with mention of com.wily.isengard.tunnelingservice

  as a key indicator.

 

 Environment

  The problem was discovered in APM 9.7.1 in a standalone Enterprise Manager running on Linux. However, this has been reproduced with Enterprise Manager running   on Windows.

  We have seen this error specifically where .NET agents are connecting to the HTTPS port of the Enterprise Manager. Connecting Java agents to the HTTPS port

  did not bring about the error.

 

  Cause

  The Enterprise Manager is an OSGi application which means it is built up of many bundles of jar files, each with their own classloader.

  Access is restricted only to specifically allowed classes as a basic principle of OSGi framework.

  Due to changes in the com.wily.isengard.com.wily.isengard.tunnelingservice_x.x.x.jar since APM 9.7, we had to add an exception to allow access

  to the org.garret.perst classes, available in the com.wily.perst_2.6.4.v20070914.jar bundle.

 

  Resolution

  To allow access to the org.garret.perst classes, do the following:

    1) Stop the Enterprise Manager

    2) Go to the Enterprise Manager's /product/enterprisemanager/configuration folder

    3) Delete all files from the folder except the config.ini file

    4) Add the following line to the config.ini file:

        org.osgi.framework.bootdelegation=org.garret.perst.*

    5) Restart the Enterprise Manager

 

Environment

Release: CEMUGD00200-9.7-Introscope to CA Application-Performance Management-Upgrade Main
Component: