ESM: Cannot generate reports fails after long time minutes with error getting status for report

Products

• CA API Gateway
• API SECURITY

Issue/Introduction

ESM on CA API Gateway 9.4 Cr03 and above a Report runs for 40 minutes and then timesout /fails with the following stack in ESM logs.

PROD ESM log will log this:

[2020/03/10-09:31:37,721]-[WARNING]-[27]-[com.l7tech.server.ems.standardreports.ReportServiceImpl]-Error getting status for report '78f9a58f-45e6-45b3-856f-537554f9e200'.

com.l7tech.server.management.api.node.ReportApi$ReportException: Report not found '78f9a58f-45e6-45b3-856f-537554f9e200'.

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshallException(JAXBEncoderDecoder.java:455)

        at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:593)

        at org.apache.cxf.jaxb.io.DataReaderImpl.read(DataReaderImpl.java:157)

        at org.apache.cxf.interceptor.ClientFaultConverter.processFaultDetail(ClientFaultConverter.java:153)

        at org.apache.cxf.interceptor.ClientFaultConverter.handleMessage(ClientFaultConverter.java:80)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)

        at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:99)

        at org.apache.cxf.binding.soap.interceptor.CheckFaultInterceptor.handleMessage(CheckFaultInterceptor.java:69)

        at org.apache.cxf.binding.soap.interceptor.CheckFaultInterceptor.handleMessage(CheckFaultInterceptor.java:34)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)

        at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:795)

        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2415)

        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2249)

        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2092)

        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)

        at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:733)

        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)

        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)

        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:544)

        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:341)

        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:294)

        at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)

        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)

        at com.sun.proxy.$Proxy165.getReportStatus(Unknown Source)

        at com.l7tech.server.ems.standardreports.ReportServiceImpl.b(Unknown Source)

        at com.l7tech.server.ems.standardreports.ReportServiceImpl.access$100(Unknown Source)

        at com.l7tech.server.ems.standardreports.d.doInTransactionWithoutResult(Unknown Source)

        at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)

        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)

        at com.l7tech.server.ems.standardreports.ReportServiceImpl.a(Unknown Source)

        at com.l7tech.server.ems.standardreports.ReportServiceImpl.access$000(Unknown Source)

        at com.l7tech.server.ems.standardreports.c.run(Unknown Source)

        at com.l7tech.server.util.ab.doRun(Unknown Source)

        at com.l7tech.server.util.ManagedTimerTask.run(Unknown Source)

        at java.util.TimerThread.mainLoop(Timer.java:555)

        at java.util.TimerThread.run(Timer.java:505)

 

Environment

Release : 9.4

Component : API GTW ENTERPRISE MANAGER

Resolution

As of Gateway 9.4 CR03 (including CR04), an update to one of the dependencies of the Gateway (specifically AdoptOpenJDK) unfortunately broke reporting functionality in ESM. This is being tracked internally as DE437057. At this time there is no ETA for a fix but it is active and being tracked. A defect was filed by Broadcom Engineering in the appropriate repository for AdoptOpenJDK's team to resolve. It looks fixed in AdoptOpenJDK 8 but it will likely be a while before we can get to that version, so we are asking the AdoptOpenJDK team to backport their fix to the version we are running with.