Layer7 container gateway 10.1 is giving Java exception under load

Products

• CA API Gateway Precision API Monitoring Module for API Gateway (Layer 7)

Issue/Introduction

We have installed layer7 10.1 container gateway on AWS EKS by following https://github.com/CAAPIM/apim-charts. We have also configured horizontal pod autoscaler. 

For container installation, we created a new image wherein we replaced the javaassist.jar with file provided in the above Broadcom article and use that image in the kubernetes deployment. But still we are getting java exceptions. Please assist us on how to resolve this issue.

 

Exceptions in logs:
{
   "exception":"java.lang.IllegalAccessError: class jdk.internal.reflect.MethodAccessorImpl loaded by javassist.Loader @xxxxxxxxx cannot access jdk/internal/reflect superclass jdk.internal.reflect.MagicAccessorImpl\n\tat java.base/java.lang.ClassLoader.defineClass1(Native Method)\n\tat java.base/java.lang.ClassLoader.defineClass(Unknown Source)\n\tat javassist.Loader.findClass(Loader.java:420)\n\tat javassist.Loader.loadClass(Loader.java:350)\n\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)\n\tat java.base/jdk.internal.misc.Unsafe.defineClass0(Native Method)\n\tat java.base/jdk.internal.misc.Unsafe.defineClass(Unknown Source)\n\tat java.base/jdk.internal.reflect.ClassDefiner.defineClass(Unknown Source)\n\tat java.base/jdk.internal.reflect.MethodAccessorGenerator$1.run(Unknown Source)\n\tat java.base/jdk.internal.reflect.MethodAccessorGenerator$1.run(Unknown Source)\n\tat java.base/java.security.AccessController.doPrivileged(Native Method)\n\tat java.base/jdk.internal.reflect.MethodAccessorGenerator.generate(Unknown Source)\n\tat java.base/jdk.internal.reflect.MethodAccessorGenerator.generateMethod(Unknown Source)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.ClientId.\u003cinit\u003e(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.SimpleClientId.\u003cinit\u003e(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapPoolManager.getLdapClient(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapClient.getInstance(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapCtx.\u003cinit\u003e(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxFromUrl(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)\n\tat java.naming/com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)\n\tat java.naming/javax.naming.spi.NamingManager.getInitialContext(Unknown Source)\n\tat java.naming/javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)\n\tat java.naming/javax.naming.InitialContext.init(Unknown Source)\n\tat java.naming/javax.naming.InitialContext.\u003cinit\u003e(Unknown Source)\n\tat java.naming/javax.naming.directory.InitialDirContext.\u003cinit\u003e(Unknown Source)\n\tat com.l7tech.server.identity.ldap.LdapIdentityProviderImpl.getBrowseContext(Unknown Source)\n\tat jdk.internal.reflect.GeneratedMethodAccessor790.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat com.l7tech.server.identity.ldap.LdapClassLoaderMethodInterceptor.invoke(Unknown Source)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)\n\tat com.sun.proxy.$Proxy204.getBrowseContext(Unknown Source)\n\tat com.l7tech.external.assertions.ldapquery.server.ServerLDAPQueryAssertion.b(Unknown Source)\n\tat com.l7tech.external.assertions.ldapquery.server.ServerLDAPQueryAssertion.doSearch(Unknown Source)\n\tat com.l7tech.external.assertions.ldapquery.server.ServerLDAPQueryAssertion.a(Unknown Source)\n\tat com.l7tech.external.assertions.ldapquery.server.ServerLDAPQueryAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)\n\tat com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.ServerPolicy.checkRequest(Unknown Source)\n\tat com.l7tech.server.policy.ak.call(Unknown Source)\n\tat com.l7tech.server.policy.ak.call(Unknown Source)\n\tat com.l7tech.common.log.HybridDiagnosticContext.doInContext(Unknown Source)\n\tat com.l7tech.server.policy.ServerPolicyHandle.checkRequest(Unknown Source)\n\tat com.l7tech.server.ar.b(Unknown Source)\n\tat com.l7tech.server.ar.a(Unknown Source)\n\tat com.l7tech.server.MessageProcessor.a(Unknown Source)\n\tat com.l7tech.server.MessageProcessor.processMessageNoAudit(Unknown Source)\n\tat com.l7tech.server.SoapMessageProcessingServlet.serviceNoAudit(Unknown Source)\n\tat com.l7tech.server.a4.call(Unknown Source)\n\tat com.l7tech.server.audit.AuditContextFactory.doWithNewAuditContext(Unknown Source)\n\tat com.l7tech.server.SoapMessageProcessingServlet.service(Unknown Source)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:770)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:742)\n\tat org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:484)\n\tat org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)\n\tat org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)\n\tat com.l7tech.server.transport.http.HttpNamespaceFilter.doFilter(Unknown Source)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat com.l7tech.server.WsdlFilter.doFilter(Unknown Source)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat com.l7tech.server.transport.http.ConnectionIdFilter.doFilter(Unknown Source)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat com.l7tech.server.transport.http.InputTimeoutFilter.doFilter(Unknown Source)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat com.l7tech.server.log.HybridDiagnosticContextServletFilter.doFilter(Unknown Source)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:100)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)\n\tat com.l7tech.server.tomcat.ResponseKillerValve.invoke(Unknown Source)\n\tat com.l7tech.server.tomcat.ConnectionIdValve.invoke(Unknown Source)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:454)\n\tat org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)\n\tat org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)\n\tat org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
   "package":"com.l7tech.server.SoapMessageProcessingServlet",
   "level":"SEVERE",
   "log":{
      "service":"myadpid [/auth]",
      "user-id":"0000000000000000ffffffffffffffff:null",
      "client-ip":"127.0.0.1",
      "request-id":"321a994be73096db-5911162efbfd7601",
      "service-folder-path":"/myADPID",
      "message":"class jdk.internal.reflect.MethodAccessorImpl loaded by javassist.Loader @xxxxxx cannot access jdk/internal/reflect superclass jdk.internal.reflect.MagicAccessorImpl",
      "user":"dmittal",
      "listen-port":"MobFed_Client"
   },
   "time":"2022-03-02T05:23:53.560-0500"
}

Environment

Release : 10.1.00_20220228

Component : CA API GATEWAY 

Resolution

The latest image will remove these java exceptions.

We performed load tests with the container gateway from the 10.1.00_20220228 image and were not seeing these exceptions any further.