CCP-Router: Java agent boot delegation failure on Liberty v17

Products

• DX APM SaaS

Issue/Introduction

I've been instrumenting our CCP-Router application. We've installed on all lower environment servers (hosted on Linux) and Liberty v20.

All good and no issues.  We tried installing on production, running Liberty v17, and having problems.

Problems with host xxx ( Liberty v17 )

Step 1: ================================ 

Using these parameters:

-Dorg.osgi.framework.bootdelegation=com.wily.* 
-Dcom.wily.introscope.agent.agentName=yyy
-javaagent:/IntroscopeAgent/wily/Agent.jar

Bootstrap.log fails with:

 [Process id=32986]     JavaAgentLauncher going to launch Java Agent
 [Process id=32983]      Exception occurred while launching Java Agent
 [Process id=32983]      Failed to lauch Java Agent. Please check logs for errors.
 [Process id=32987]      Exception occurred while launching Java Agent
 [Process id=32987]      Failed to lauch Java Agent. Please check logs for errors.
 [Process id=32986]      Exception occurred while launching Java Agent
 [Process id=32986]      Failed to lauch Java Agent. Please check logs for errors.

Step 2: ================================ 

I tried bypassing Bootstrap agent by pointing the javaagent to the main agent jar and providing the profile location (no other changes):

-javaagent:/IntroscopeAgent/wily/releases/2022.11/Agent.jar
-Dcom.wily.introscope.agentProfile=/IntroscopeAgent/wily/releases/2022.11/core/config/IntroscopeAgent.profile 

Failed with:

[ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to create the delegate agent

Step 3: ================================ 

I tried removing the boot delegation line because it's not in the current documentation:

Failed as above...

Step 4: ================================ 

I searched my notes and tried changing boot delegation to which has worked in other instances:

-Dorg.osgi.framework.bootdelegation=com.wily.*,sun.*,com.sun.*,com.ibm.*

Failed with:

 [ERROR] [IntroscopeAgent.Agent] A problem occurred while attempting to create the delegate agent
Exception in thread "main" java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
 at java.lang.reflect.Method.invoke(Method.java:508)
 at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:408)
 at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:423)
Caused by: java.lang.NoSuchMethodError: agent/main/AgentShim$DesperationTracer.<init>(Lagent/main/AgentShim$1;)V (loaded from /usr/WebSphere/wlp/java/8.0/jre/jre/lib/rt.jar by <Bootstrap Loader>) called from class agent.main.AgentShim (loaded from /IntroscopeAgent/wily/releases/2022.11/Agent.jar by <Bootstrap Loader>).
 at agent.main.AgentShim.<clinit>(AgentShim.java:133)
 at com.wily.introscope.api.instrument.JavaAgent.initializeAgent(JavaAgent.java:275)
 at com.wily.introscope.api.instrument.JavaAgent.premain(JavaAgent.java:117)
 ... 6 more

Fatal error: processing of -javaagent failed

Step 5: ================================

We installed on server xxx, also production. (with liberty version 20.0.0.12 ) and all worked fine...

==========================

Java Version information for xxx ( Liberty v17 ):

[yyy]$ /usr/WebSphere/wlp/java/8.0/jre/bin/java -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build pxl6480sr4fp10-20170727_01(SR4 FP10))
IBM J9 VM (build 2.8, JRE 1.8.0 Linux ppc64le-64 Compressed References 20170722_357405 (JIT enabled, AOT enabled)
J9VM - R28_20170722_0201_B357405
JIT  - tr.r14.java_20170722_357405
GC   - R28_20170722_0201_B357405_CMPRSS
J9CL - 20170722_357405)
JCL - 20170726_01 based on Oracle jdk8u144-b01

 

Environment

Release : SAAS

Resolution

Closed due to non-response

On boot delegation, this may help

Caused by the feature in WebSphere class caches. As the DX SaaS Agent performs bytecode instrumentation, 
refers to the Java Agent code are also stored in this cache. Which is causing the problem and creating the above ERROR.

Resolution
Add this JVM argument to the startup script. 

-Xshareclasses:none

Save the file 
Start the Application.