Workstation suddenly crashing when clicking on Errors or Traces tabs in Investigator

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

Whenever we click on the Errors Or Traces tab in Investigator window of java client (local worktation install or webstart worktation), the client crashes out and shuts down. The EM log is showing following ERROR message.

[ERROR] [Workstation.MessageServiceClient] Exception: com.wily.util.exception.UnexpectedExceptionError: com.wily.introscope.spec.server.QueryException: Could not query: read past EOF calling method: com.wily.isengard.messageprimitives.service.CompressedMessageServiceCallMessage: {com.wily.introscope.spec.server.beans.event.IEventManager.executeEventQuery, v1, [[SessionToken|key: 4699696045093428512, po: Workstation_216.client_main], [email protected]], source: Workstation_216.client_main:12442, remoteHost: {Unknown}}
com.wily.isengard.messageprimitives.ConnectionException
at com.wily.isengard.messageprimitives.service.MessageServiceClient.sendRequest(MessageServiceClient.java:197)
at com.wily.isengard.messageprimitives.service.MessageServiceClient.invoke(MessageServiceClient.java:359)
at com.sun.proxy.$Proxy47.executeEventQuery(Unknown Source)
at com.wily.introscope.console.thornhill.ui.transactiontrace.typeview.AEventTypeViewer.queryForTransactions(AEventTypeViewer.java:365)
at com.wily.introscope.console.thornhill.ui.transactiontrace.typeview.AEventTypeViewer.access$8(AEventTypeViewer.java:325)
at com.wily.introscope.console.thornhill.ui.transactiontrace.typeview.AEventTypeViewer$2.IEventMarkedRunnable_run(AEventTypeViewer.java:241)
at com.wily.util.event.EventPump.immediateRun(EventPump.java:218)
at com.wily.util.event.EventPump.access$7(EventPump.java:214)
at com.wily.util.event.EventPump$Posting.deliver(EventPump.java:608)
at com.wily.util.event.EventPump$EventPumpThread.run(EventPump.java:827)
Caused by: com.wily.util.exception.UnexpectedExceptionError: com.wily.introscope.spec.server.QueryException: Could not query: read past EOF
at com.wily.introscope.server.enterprise.entity.transactiontrace.ClusteredEventManager.doExecuteEventQuery(ClusteredEventManager.java:522)
at com.wily.introscope.server.enterprise.entity.transactiontrace.AEventManager.executeEventQuery(AEventManager.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.wily.isengard.messageprimitives.MethodCallUtilities.callInterface(MethodCallUtilities.java:75)
at com.wily.isengard.messageprimitives.MethodCallUtilities.callInterface(MethodCallUtilities.java:29)
at com.wily.isengard.messageprimitives.service.MessageService.attemptMethodCall(MessageService.java:183)
at com.wily.isengard.messageprimitives.service.MessageService.handleMethodCallMessage(MessageService.java:135)
at com.wily.isengard.messageprimitives.service.MessageService.receiveMessage(MessageService.java:161)
at com.wily.isengard.postoffice.Mailbox.handleMessage(Mailbox.java:252)
at com.wily.isengard.postoffice.PostOffice.deliverInternal(PostOffice.java:532)
at com.wily.isengard.postoffice.PostOffice.access$2(PostOffice.java:478)
at com.wily.isengard.postoffice.PostOffice$DeliveryItem.run(PostOffice.java:886)
at com.wily.EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:728)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.wily.util.exception.UnexpectedExceptionError: com.wily.introscope.spec.server.QueryException: Could not query: read past EOF
at com.wily.introscope.server.enterprise.entity.transactiontrace.ClusteredEventManager.doExecuteEventQuery(ClusteredEventManager.java:512)
... 16 more
Caused by: java.util.concurrent.ExecutionException: com.wily.introscope.spec.server.QueryException: Could not query: read past EOF
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.wily.introscope.server.enterprise.entity.transactiontrace.ClusteredEventManager.doExecuteEventQuery(ClusteredEventManager.java:487)
... 16 more
Caused by: com.wily.introscope.spec.server.QueryException: Could not query: read past EOF
at com.wily.introscope.server.enterprise.entity.transactiontrace.EventManager.doExecuteEventQuery(EventManager.java:152)
at com.wily.introscope.server.enterprise.entity.transactiontrace.EventManager.doExecuteEventQuery(EventManager.java:239)
at com.wily.introscope.server.enterprise.entity.transactiontrace.AEventManager.executeEventQuery(AEventManager.java:123)
at sun.reflect.GeneratedMethodAccessor1080.invoke(Unknown Source)
... 13 more
 [INFO] [Workstation] Beginning orderly shutdown...
[INFO] [Workstation.Login] Logged out user "myuser"
[INFO] [Workstation] Orderly shutdown complete.

Environment

Release : 10.7.0

Component : Introscope

Resolution

Stop EM, remove old trace files under traces directory (older then 30 days), run the indexer from under /opt/apm/introscope/tools

Followed the process in this KB doc: 

https://knowledge.broadcom.com/external/article/29182/em-logs-filled-with-this-error-error-luc.html

Can go in to the traces and errors tabs without workstation crashing