DLP agents with unknown connection status

Products

• Data Loss Prevention
• Data Loss Prevention Endpoint Prevent
• Data Loss Prevention Endpoint Suite

Issue/Introduction

Many agents in the environment are showing an unknown connection status, and the list appears to be growing.

 

 

Cause

In the logs, we found

WARNING: Rejecting existing connection with AgentId 'XXXXXXXXX' since a new connection with the same AgentId has come in.

This indicates the agent is not able to complete its communication to the endpoint server.

If the server does not receive a completed communication it sets the agent status to unknown until the communication is completed.

In this case the agents are unable to complete the communication before the communication was being started over again.

Resolution

The resolution has two parts.

• Increase the polling interval to be large enough to give the agent time to complete its communication before it starts another. **

• OutOfMemory errors were observed in the aggregator log, and Java heap settings were still at the default values on the detection server.  These may need to be increased depending on the environment and the policies in effect.  Increase the settings to match the physical hardware on the server.* This allows the server to process the information faster. Increase the java heap on the endpoint server and restart the detection server services, increase the polling in the agent configuration and push the change to the agents. Many of the agents should immediately start to show as reporting and all of the agents return to normal operation over the next couple of days as more and more of the agents start to slow down and complete their communications.

* Please note, as of 15.8 much of the detection components have been moved to native memory, not Java heap. So the need to increase this value should not be as common, nor should the amount be as much. typically a max of 1 GB should be more than sufficient. Increasing this too much can take away from native memory and cause further issues. But do check for OOM errors in the file reader log, and if so the change can be done in the advanced settings of the detection server in the Enforce UI. A good test is to leave the minimum at the default of 128 MB and set the max very high, restart the services, and watch the actual memory consumption. You can then set the max value a little higher than the max you see taken by the service. Always be sure to leave enough for the OS, native memory function, and file reader.

Additional Information

• ** The number of endpoint computers exceeds the number recommended for a single endpoint server in relation to the standard polling interval. "The frequency in which the agent will attempt to communicate to the endpoint server."  The rule of thumb being 1 minute for every one thousand endpoints. "Please note, this is a variable which is dependent upon your environment."
• Both of these settings would be considered environment tuning. Our professional services division can assist you in the exact settings that would be required for your environment, and the policies you have in use.