RCA for timeout errors for the DA pushing events to EM

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

We are noticing the following errors in the DA karaf.log and would like to look into their root cause:

RROR | EventPush        | 2022-10-24T18:06:56,044 | EventProducerWS | or.eventproducer.EventProducerWS  997 | a.im.NPCConnector.bundle |       | Failed to push 1200 events: Marshalling Error: connect timed out
INFO  | EventPush        | 2022-10-24T18:06:56,084 | ReflectionServiceFactoryBean | ory.ReflectionServiceFactoryBean  436 | g.apache.cxf.cxf-rt-wsdl |       | Creating Service {http://netqos.com/nqevents/EventManager}IEventManagerWSService from class com.ca.im.portal.api.datasources.interfaces.nqevents.eventmanagerws.IEventManagerWS
INFO  | nstance_Worker-4 | 2022-10-24T18:07:02,275 | BatchProcessorImpl | ocessorImpl$BatchTriggerListener  243 | .ca.im.aggregator.loader |       | Batch process job: GroupsETLJob completed in 0:00:02.274( 2274 ms )
WARN  | EventPush        | 2022-10-24T18:07:26,125 | PhaseInterceptorChain | ache.cxf.common.logging.LogUtils  465 | org.apache.cxf.cxf-core |       | Interceptor for {http://netqos.com/nqevents/EventManager}IEventManagerWSService#{http://netqos.com/nqevents/EventManager}Push has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Marshalling Error: connect timed out
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:267) ~[bundleFile:3.5.0]
        at org.apache.cxf.jaxb.io.DataWriterImpl.write(DataWriterImpl.java:239) ~[bundleFile:3.5.0]
        at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:118) ~[bundleFile:3.5.0]
        at org.apache.cxf.wsdl.interceptors.BareOutInterceptor.handleMessage(BareOutInterceptor.java:68) ~[bundleFile:3.5.0]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) [bundleFile:3.5.0]
        at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528) [bundleFile:3.5.0]
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:439) [bundleFile:3.5.0]
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:354) [bundleFile:3.5.0]
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:312) [bundleFile:3.5.0]
        at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96) [bundleFile:3.5.0]
        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140) [bundleFile:3.5.0]
        at com.sun.proxy.$Proxy416.push(Unknown Source) [?:?]
        at com.ca.im.connector.eventproducer.EventProducerWS.pushEvents(EventProducerWS.java:985) [bundleFile:?]
        at com.ca.im.connector.eventproducer.EventProducerWS$1.run(EventProducerWS.java:429) [bundleFile:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: javax.xml.bind.MarshalException
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:303) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:226) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:80) ~[jakarta.xml.bind-api-2.3.3.jar:2.3.3]
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.writeObject(JAXBEncoderDecoder.java:642) ~[bundleFile:3.5.0]
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:244) ~[bundleFile:3.5.0]
        ... 14 more
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:?]
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:?]
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?]
        at java.net.Socket.connect(Unknown Source) ~[?:?]
        at sun.net.NetworkClient.doConnect(Unknown Source) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(Unknown Source) ~[?:?]
        at sun.net.www.http.HttpClient.New(Unknown Source) ~[?:?]
        at sun.net.www.http.HttpClient.New(Unknown Source) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source) ~[?:?]
        at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:274) ~[bundleFile:3.5.0]
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343) ~[bundleFile:3.5.0]
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304) ~[bundleFile:3.5.0]
        at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:307) ~[bundleFile:3.5.0]
        at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) ~[bundleFile:3.5.0]
        at org.apache.cxf.io.AbstractThresholdOutputStream.unBuffer(AbstractThresholdOutputStream.java:89) ~[bundleFile:3.5.0]
        at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:63) ~[bundleFile:3.5.0]

  at com.sun.xml.bind.v2.runtime.output.Encoded.write(Encoded.java:137) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.writeName(UTF8XmlOutput.java:211) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.beginStartTag(UTF8XmlOutput.java:139) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.XMLSerializer.leafElement(XMLSerializer.java:294) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$StringImplImpl.writeLeafElement(RuntimeBuiltinLeafInfoImpl.java:1040) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$StringImplImpl.writeLeafElement(RuntimeBuiltinLeafInfoImpl.java:1019) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.reflect.TransducedAccessor$CompositeTransducedAccessorImpl.writeLeafElement(TransducedAccessor.java:226) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty.serializeBody(SingleElementLeafProperty.java:100) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:329) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:666) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.serializeItem(ArrayElementNodeProperty.java:39) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:142) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:129) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:329) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:666) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.serializeBody(SingleElementNodeProperty.java:128) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:329) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:666) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.serializeItem(ArrayElementNodeProperty.java:39) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:142) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:129) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:329) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:666) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.serializeBody(SingleElementNodeProperty.java:128) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:329) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:666) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl$1.serializeBody(ElementBeanInfoImpl.java:125) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl$1.serializeBody(ElementBeanInfoImpl.java:100) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl.serializeBody(ElementBeanInfoImpl.java:302) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl.serializeRoot(ElementBeanInfoImpl.java:309) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.ElementBeanInfoImpl.serializeRoot(ElementBeanInfoImpl.java:45) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:464) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:298) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:226) ~[jaxb-runtime-2.3.3.jar:2.3.3]
        at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:80) ~[jakarta.xml.bind-api-2.3.3.jar:2.3.3]
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.writeObject(JAXBEncoderDecoder.java:642) ~[bundleFile:3.5.0]
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:244) ~[bundleFile:3.5.0]
        ... 14 more
ERROR | EventPush        | 2022-10-24T18:07:26,126 | EventProducerWS | or.eventproducer.EventProducerWS  997 | a.im.NPCConnector.bundle |       | Failed to push 1200 events: Marshalling Error: connect timed out
INFO  | EventPush        | 2022-10-24T18:07:26,136 | ReflectionServiceFactoryBean | ory.ReflectionServiceFactoryBean  436 | g.apache.cxf.cxf-rt-wsdl |       | Creating Service {http://netqos.com/nqevents/EventManager}IEventManagerWSService from class com.ca.im.portal.api.datasources.interfaces.nqevents.eventmanagerws.IEventManagerWS
WARN  | EventPush        | 2022-10-24T18:07:56,173 | PhaseInterceptorChain | ache.cxf.common.logging.LogUtils  465 | org.apache.cxf.cxf-core |       | Interceptor for {http://netqos.com/nqevents/EventManager}IEventManagerWSService#{http://netqos.com/nqevents/EventManager}Push has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Marshalling Error: connect timed out
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:267) ~[bundleFile:3.5.0]
        at org.apache.cxf.jaxb.io.DataWriterImpl.write(DataWriterImpl.java:239) ~[bundleFile:3.5.0]
        at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:118) ~[bundleFile:3.5.0]
        at org.apache.cxf.wsdl.interceptors.BareOutInterceptor.handleMessage(BareOutInterceptor.java:68) ~[bundleFile:3.5.0]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) [bundleFile:3.5.0]
        at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528) [bundleFile:3.5.0]

 

Environment

Release : 22.2

Cause

In this case the 3rd party component JaxWS (Which is responsible for making the connection to EM) was using the IPV4 address and not the IPV6 address in the /etc/hosts file 

Resolution

Commenting out the IPV4 from the /etc/hosts file resolved the issue.