Wensphere Agent will not start or generate log files.

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We have an appserver running websphere where the apm 11 javaagent argument shows in the process list, however it does not start.  There are no log files and agent directory permissions look okay 644 for files and 755 for directories..  This same agent has been deployed to another server  where it functions. Our local IBM WAS has been unable to remediate with APM admin assistance.  

stackdump websphere log

[11:23 AM] XXXX (IBM) (Guest)
[6/17/21 11:14:34:068 EDT] 00000001 CustomService W   XXXX: Unable to create CustomService, Introscope Custom Service
java.lang.NoClassDefFoundError: com.wily.introscope.agent.service.IAgentService
        at java.lang.J9VMInternals.newInstanceImpl(Native Method)
        at java.lang.Class.newInstance(Class.java:1852)
        at com.ibm.ws.runtime.component.CustomServiceImpl.initialize(CustomServiceImpl.java:113)
        at com.ibm.ws.runtime.component.ContainerHelper.initWsComponent(ContainerHelper.java:1193)
        at com.ibm.ws.runtime.component.ContainerHelper.initializeComponent(ContainerHelper.java:1100)
        at com.ibm.ws.runtime.component.ContainerHelper.initializeComponents(ContainerHelper.java:929)
        at com.ibm.ws.runtime.component.ContainerImpl.initializeComponents(ContainerImpl.java:776)
        at com.ibm.ws.runtime.component.ContainerImpl.initializeComponents(ContainerImpl.java:750)
        at com.ibm.ws.runtime.component.ServerImpl.initialize(ServerImpl.java:370)
        at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:307)
        at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:238)
        at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:711)
        at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
        at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:96)
        at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
        at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
        at org.eclipse.core.launcher.Main.run(Main.java:981)
        at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:424)
        at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:177)
Caused by: java.lang.ClassNotFoundException: com.wily.introscope.agent.service.IAgentService
        at java.net.URLClassLoader.findClass(URLClassLoader.java:610)
        at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:244)
        at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:944)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:889)
        at com.ibm.ws.bootstrap.ExtClassLoader.loadClass(ExtClassLoader.java:135)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:872)
        ... 39 more 

Environment

Release : 20.2

Component : APM Agent

Resolution

APM agent don't appear to be instrumented.

Further examination of the system show it already have dynatrace oneagent running.

Upon removal of /opt/dynatrace and /opt/dtrun directories, cycling of websphere, Application JVMs started without issues.