Gateway v10.1 incompatibility with AppDynamics - Error starting server : access denied ("java.util.PropertyPermission"

Products

• CA API Gateway

Issue/Introduction

We have been using AppDynamics along with CA API Gateway v10.0 for some time now without any issues.

When we upgraded the Gateways to v10.1 w/AppDynamics the Gateway's ssg service stopped working. 

During troubleshooting the only solution found was to disable AppDynamics.  It was discovered that the issue was with java. 

We opened a ticket with AppDynamics and with customer support did the following:

Troubleshooting steps Performed with AppDynamics vendor support:

1. Updated the ssg.policy file with new grant permissions for the APM app agent.
2. Updated the java.policy file with new grant permissions for the APM app agent.
3. Added the following options to the Java startup options in the node.properties file: "-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true --add-reads jdk.jfr=ALL-UNNAMED"
4. Tested starting the Gateway services using a blank "fake" agent jar file by updating the Java startup options to call the blank jar file instead of the real agent file. Even when calling this blank jar file, we still receive the same java permission denied errors.

Jul 18, 2022 4:26:10 PM com.l7tech.server.boot.GatewayMain main
WARNING: Error starting server : access denied ("java.util.PropertyPermission" "com.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize" "read")
java.security.AccessControlException: access denied ("java.util.PropertyPermission" "com.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize" "read")

 

Environment

Release : 10.1

Component : API GATEWAY

Cause

The JVM profilers used by APM tools are intrusive by definition , security permissions are needed 

Resolution

AppDynamics AppAgent (Java Agent) w/XML Gateway v10.1 Fix

Required components and versions:

Layer7 API Gateway version 10.1 CR01

Layer7 API Gateway internal database upgraded to 10.1 schema

openjdk version "11.0.13" 2021-10-19

OpenJDK Runtime Environment Temurin-11.0.13+8 (build 11.0.13+8)

OpenJDK 64-Bit Server VM Temurin-11.0.13+8 (build 11.0.13+8, mixed mode)

Steps:

1. mysqladmin stop-replica (Nodes 1 and 2 only)
2. service ssg stop
3. systemctl stop appdynamics-machine-agent
4. cp /opt/SecureSpan/Gateway/runtime/etc/profile.d/sh /opt/SecureSpan/Gateway/runtime/etc/profile.d/ssgruntimedefs.backup
5. cp /opt/SecureSpan/Gateway/runtime/etc/policy /opt/SecureSpan/Gateway/runtime/etc/ssg.policy.backup
6. vi cp /opt/SecureSpan/Gateway/runtime/etc/profile.d/sh
7. Add:       
   
   default_java_opts="$default_java_opts -javaagent:/opt/app/appdynamics/app-agent/javaagent.jar"

        default_java_opts="$default_java_opts -Dappdynamics.agent.tierName=<tier-name>"

        default_java_opts="$default_java_opts -Dappdynamics.agent.nodeName=<hostname>"

        default_java_opts="$default_java_opts -Dappagent.start.timeout=1"

        default_java_opts="$default_java_opts -Dappdynamics.delegate.parent.classes=org.w3c.*,org.apache.xerces.*,org.xml.*,javax.xml.*"

        default_java_opts="$default_java_opts -Dappagent.usebootstrap.as.parent=false"

        default_java_opts="$default_java_opts -Dappdynamics.agent.use.stderr=file:/opt/app/appdynamics/app-agent/agent_boot.log"

        default_java_opts="$default_java_opts -Dappdynamics.agent.use.agent.classloader.context=false"

        #default_java_opts="$default_java_opts -Dappdynamics.agent.log4j2.disabled=true"

        #default_java_opts="$default_java_opts --add-reads java.xml=ALL-UNNAMED"

8. vi /opt/SecureSpan/Gateway/runtime/etc/policy
   
   a) First grant in file has permission java.security.AllPermission;
   b) Add: grant codeBase "file:/opt/app/appdynamics/app-agent/-" {

   permission java.security.AllPermission;

   permission java.lang.RuntimePermission "getClassLoader";

};

9. Restart Gateway Service :
   Service ssg restart
   
   
10. Start Appdynamics agent:
    systemctl start appdynamics-machine-agent
    
    
11. Start Replication
    mysqladmin start-replica (Nodes 1 and 2 only)
    
    
12. Verify if replication is up. (Nodes 1 and 2 only)
    
    
13. From the ssg Menu verify if ssg service is running.
     
    2) Display Layer7 API Gateway configuration menu
    then
    7) Manage Layer7 API Gateway status
    
    
    
14. Verify from AppDynamics dashboard if AppDynamics is ingesting data.

 

 

 

Additional Information

API Gateway: Support for third-party applications/agents and/or integrating with third-party tools

https://knowledge.broadcom.com/external/article?articleId=16296