APM - Unable to instrument Atlasian JIRA, exception " Constructor threw exception; nested exception is java.lang.VerifyError: Inconsistent stackmap frames at branch target '

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We have updated our Jira version to v8.20.13 which includes a jdk 11 + Plugin Adaptavist ScriptRunner for JIRA, versión: 6.49.0

We are using recommended setting as per documentation:

JAVA_OPTS="$JAVA_OPTS -javaagent:$AGENT_HOME/Agent.jar -noverify -verbose:class -Datlassian.org.osgi.framework.bootdelegation.extra=com.wily,com.wily.*,sun.*,org.w3c.*,org.xml.sax.*,org.apache.xerces.*,com.sun.org.apache.xalan.*,org.apache.xml,org.apache.xml.*,javax.xml,javax.xml.*,com.sun,com.sun.*org.apache.xalan,org.apache.xalan.* -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl"

Introscope EM version: 10.7

When trying to to start the plugin we get :

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'scriptFieldConfigProvider': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'scriptFieldConfigurationServiceImpl': Unsatisfied dependency expressed through constructor parameter 4; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'scriptFieldTypeManager': Unsatisfied dependency expressed through constructor parameter 3; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'issueParametersCapturingImmutableCustomFieldFactory': Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'temporaryIssueService': Unsatisfied dependency expressed through constructor parameter 6; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'workflowUtil': Unsatisfied dependency expressed through constructor parameter 10; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scriptRunnerImpl': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.onresolve.scriptrunner.runner.ScriptRunnerImpl]: Constructor threw exception; nested exception is java.lang.VerifyError: Inconsistent stackmap frames at branch target 397

Adding "-noverify" as below doesn't help

-javaagent:$AGENT_HOME/Agent.jar -noverify -verbose:class -Datlassian.org.osgi.framework.bootdelegation.extra=com.wily,com.wily.*,sun.*,org.w3c.*,org.xml.sax.*,org.apache.xerces.*,com.sun.org.apache.xalan.*,org.apache.xml,org.apache.xml.*,javax.xml,javax.xml.*,com.sun,com.sun.*org.apache.xalan,org.apache.xalan.* -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl"

It still does not start and we get 

Unsatisfied dependency expressed through constructor parameter 10; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scriptRunnerImpl': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.onresolve.scriptrunner.runner.ScriptRunnerImpl]: Constructor threw exception; nested exception is java.lang.NullPointerException
 

Environment

Release : 10.7 and SaaS Agents

JIRA version 8.20.13 + JIRA Plugin Adaptavist ScriptRunner for JIRA, versión: 6.49.0

Resolution

Recommendations:

- Upgrade to : 10.8 Agent version (available from the Support Download site) or latest Saas Agent version (contact Broadcom Support to obtain a copy)

- Reduce instrumentation as below:

1) stop the jvm

2) delete all existing logs

3) reduce instrumentation as below:

a) Set introscope.agent.reduceAgentMemoryOverhead=true

b) disable Smart Instrumentation:
introscope.agent.deep.entrypoint.enabled=false
introscope.agent.deep.instrumentation.enabled=false
introscope.agent.deep.trace.enabled=false
introscope.agent.deep.errorsnapshot.enable=false
introscope.agent.deep.automatic.trace.crossprocess.enabled=false
introscope.agent.automatic.backend.detection.enabled=false

IMPORTANT: delete AutoPersist.pbd file from config/hotdeploy folder

g) Disable socket instrumentation:

If you are using APM SaaS Agent: Turn off the socket tracers in acc-master-toggles.pbd file as below:

#TurnOn: SocketTracing
#TurnOn: NIOSocketTracing
#TurnOn: NIOSocketSummaryTracing
#TurnOn: NIOSelectorTracing
#TurnOn: NIODatagramTracing
#TurnOn: NIODatagramSummaryTracing

h) Disable header insertion.

com.wily.introscope.agent.soapheaderread.enabled=false
com.wily.introscope.agent.soapheaderinsertion.enabled=false
com.wily.introscope.agent.httpheaderread.enabled=false
com.wily.introscope.agent.httpheaderinsertion.enabled=false

4) start the jvm 

If the problem persists: stop the jvm and collect the jira and agent logs

If the problem doesn't persists: you can try to  enable above disabled features 1 by 1 until you identify which exact feature causing the issue

Additional Information

For additional Java Agent Troubleshooting steps refer to

https://knowledge.broadcom.com/external/article/145354/apm-introscope-java-agent-troubleshooti.html