Causes of Workstation Connection Failures and Agent Disconnections.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

 Unable to connect to Workstation or the Agent is disconnecting. Also, the MOM log is showing the below error:

ERROR MESSAGE:

[ERROR] [Acceptor Helper 2] [Manager.Transport] Incoming connection failed because of exception
java.lang.RuntimeException: Transporting data too big consumed 4001 buffers exceeded max allowed: 4000
at com.wily.isengard.postofficehub.link.v1.server.BufferPool.take(BufferPool.java:100)
at com.wily.isengard.postofficehub.link.v1.server.ByteBufferInputStream.readFromChannel2(ByteBufferInputStream.java:224)
at com.wily.isengard.postofficehub.link.v1.server.ByteBufferInputStream.readFromChannel(ByteBufferInputStream.java:188)
at com.wily.isengard.postofficehub.link.v1.server.ByteBufferInputStream.read(ByteBufferInputStream.java:138)
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337) 

at java.io.DataInputStream.readUTF(DataInputStream.java:589) 

at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at com.wily.isengard.postofficehub.link.v1.server.ABaseServerTransport.accept(ABaseServerTransport.java:114)
at com.wily.isengard.postofficehub.link.server.AcceptorHelper.run(AcceptorHelper.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

Environment

APM 9.7, 10.0, 10.2, 10.3, 10.5, 10.5.x

Cause

 - For EM as a server, this error could occur because a client is sending huge amounts of data. The client could be an agent or workstation sending data to the EM.

 - Prior to 9.7.x, in this situation an EM could crash with an Out Of Memory exception. In 9.7.x and later, a clamp was added to prevent the crash. Once the clamp is hit, then the EM will disconnect the client, and the client could try to re-connect automatically.

 - The impact of increasing this clamp is that the EM needs to have enough memory/heap.

 - To calculate approximately how much memory the value set will require, please use the following formula (128K * VALUE), e.g.:

            128K * 8000 = 1GB this is extra for the input buffers of transportation layer

Resolution

 Open the "IntroscopeEnterpriseManager.properties" in the MOM EM's <EM_HOME>\config folder and add the following property:

 transport.buffer.input.maxNum=8000

or

transport.buffer.input.maxNumNio=8000 if using NIO (default on Linux)

 

 A restart of the EM is needed for this new setting to take effect.

Additional Information

See TEC1284029 for information on how to block connections from these agents