Error 3900: Internal Communications error received from Endpoint Server.
search cancel

Error 3900: Internal Communications error received from Endpoint Server.

book

Article ID: 170745

calendar_today

Updated On:

Products

Data Loss Prevention Enforce

Issue/Introduction

System alerts generated in the Enforce console for an error with the connection to the DLP Endpoint server:

"Internal communications error. Please see Aggregator.log for errors. Search for the string TC - Unexpected Exception"

The aggregator log provides detail information on the agent connection.

File: Endpoint_Server\logs\debug\Aggregator5.log
Date: 11/2/2017 6:40:50 PM
Class: com.symantec.dlp.communications.common.activitylogging.JavaLoggerImpl
Method: log
Level: SEVERE
Message:
java.lang.IllegalStateException: SSLEngine already closed
    at org.jboss.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1074)
    at org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:623)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:292)
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:254)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at com.symantec.dlp.communications.transportlayer.impl.NettyTransportConnection$WriteOutboundDataTask.run(NettyTransportConnection.java:1588)
    at com.symantec.dlp.communications.transportlayer.impl.PrioritizedTaskQueue.run(PrioritizedTaskQueue.java:74)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
TC - Unexpected exception  for connection number 1898, '<endpoint agent>' at 2017-11-02 06:40:50. Write failed. Connection statistics

Cause

The expected behavior of agent communication is as follows: (relevant advanced agent settings in parenthesis): 

  • Agent checks in based on it's polling interval(ServerCommunicator.CONNECT_POLLING_INTERVAL_SECONDS.int)
  • Agent communicates all data and then the server connection remains idle
  • After 30 seconds the server disconnects the agent gracefully.(EndpointCommunications.IDLE_TIMEOUT_IN_SECONDS.int)
  • Agent waits it's polling interval to connect again 


When the server didn't gracefully close the connection. We generate a system event upon unexpected connection closures.

Common causes include: 

  • Remote Host Disconnection: Users switching wireless networks, users closing their laptop lid while the agent is connected to the endpoint server. 
  • Firewall/Load Balancer Timeouts: A firewall or load balancer terminating idle or long‐lived connections.
  • Load balancer Health-checks: Since a health-check will most likely not keep the connection active until the server gracefully disconnects the session, a network probe can cause the alert.
  • Vulnerability/port scanners: Security scans are likely just checking for open ports and do not keep the connection open until the server gracefully disconnects the session. 
  • NAT/Proxy Reset: A network address translation device or proxy device dropping or resetting a connection due to session expiry.
  • Network Congestion or Interruption: Severe network latency or packet loss that forces a timeout and disconnection.
  • External Infrastructure Maintenance: Scheduled maintenance or an unexpected outage on intermediate network equipment (like routers or switches) that forces connections to be cut.
  • Agent misconfiguration: An agent misconfiguration such as a short polling interval or misaligned heartbeat interval can cause the alert. 

Resolution

In most cases, 3900s can be ignored. Furthermore, they can be difficult to track post-mortem because something as simple as a closed laptop lid can trigger a 3900.  Identifying when and where such a thing occurred can be difficult or impossible. 

As long as the agents are still receiving configuration and policy updates, reporting incidents, and completing agent tasks such as 'pull logs', there is likely no systemic issue to pursue. 
If policy updates, configurations, or troubleshooting tasks appear delayed or non-functional, then additional efforts are warranted. 

This is possibly caused by a misconfiguration in the agent configuration.  Try creating a new agent config and leaving the advanced agent configuration as close to the default as possible. See DLP Agent status not reporting as expected on Enforce for details on how an agent may be misconfigured causing connection issues.  

 

 

Powered by Wolken Software