The Endpoint Server may hang intermittently, and appear not to report any new incidents.

book

Article ID: 159551

calendar_today

Updated On:

Products

Data Loss Prevention Endpoint Prevent

Issue/Introduction

The Endpoint Server may hang intermittently and appear not to report any new incidents. This issue may occur even if the server appears to be running as indicated (by a green arrow) on the System Overview page.

 

The IncidentWriter*.log may show some of the following log entries:   

 

SEVERE: Error sending incident. Unexpected error occurred while sending an incident.

Receiving side could not open file for transfer of data !

Look in the incident writer log for more information.

Mar 28, 2010 11:21:35 AM com.vontu.communication.dataflow.ShippingTask processErrorResponse

WARNING: Shipping Task(282297): Receiving side could not open file for transfer of data !

Mar 28, 2010 11:22:35 AM com.vontu.communication.dataflow.ShippingTask processErrorResponse

WARNING: Shipping Task(282304): Receiving side could not open file for transfer o

 

If the Endpoint Server is restarted, the incidents are pushed to the Enforce Server and reported.

Environment

Please note: Details in this TECHNOTE do not apply to default setup for versions of DLP 12.5 and after.

All content in this TECHNOTE regarding persistent connections only applies to DLP versions 12.0.1 and earlier.

Resolution

Symptoms:

 

In the UI the Endpoint Server appears up and running (green arrow in system overview). The only indicator of an issue is that no incidents are processed and pushed to the Enforce Server. The System Overview page shows that no new incidents are reported. Only when the Endpoint Server is restarted are the incidents pushed to Enforce.

 

The IncidentWriter*.log may show some of the following log entries.

 

SEVERE: Error sending incident. Unexpected error occurred while sending an incident.

Receiving side could not open file for transfer of data !

Look in the incident writer log for more information.

Mar 28, 2010 11:21:35 AM com.vontu.communication.dataflow.ShippingTask processErrorResponse

WARNING: Shipping Task(282297): Receiving side could not open file for transfer of data !

Mar 28, 2010 11:22:35 AM com.vontu.communication.dataflow.ShippingTask processErrorResponse

WARNING: Shipping Task(282304): Receiving side could not open file for transfer o

 

When the Vontu Services are restarted everything clears up and the Endpoint Server collects incidents.

 

Cause:

 

An ideal Endpoint Server deployment relies on two things:

- A reliable network (no dropped connection)

- The network supports inactive and active persistent connections. This functionality means that inactive connections are not dropped.

 

In a real word deployment, proxies, gateways, firewall and so forth that may drop connections when they are idle for a certain time. Or network instability can cause these connections to drop.

 

If a connection drop occurs, the connection is re-established. However, the OS holds half-open connections for some time. Active proxies and gateways can introduce lag times by holding the connection to the Endpoint Server open after the connection to the Endpoint agent dropped.

This issue ultimately results in a higher number of connections to the Endpoint Server.

Once the number of open ports to the Endpoint Server reaches the OS limit, the Endpoint Server is unable to create and push incidents to the Enforce Server.
The OS limit is found in; (KEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersMaxUserPort ).

 

 

Suggestion A :

 

1) Within the Vontu/protect/config/Aggregator.properties set modify following values.

 

FROM

# The maximum number of simultaneously connected agents

maxConnections = 10000

 

TO

# The maximum number of simultaneously connected agents

maxConnections = 1000

 

Note: These are initial changes and you may need to monitor the environment if further adjustment is needed. This setting reduces this consumption of sockets to limit the number of simultaneous connections. This setting causes the server to reject connections after some point and the agents back off for some time. You may have to reduce the number to as low as 1000 connections. So that at any given time a maximum of 1000 agents can be connected.

 

Note also that this setting has to be made with the overall numbers of agents in mind. You can also choose a lower number, such as 500 if the number of agents allows it.

 

 

2) Within the Registry , modify the value for; HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersMaxUserPort to 65000

Note : Valid Range: 5000-65534 (decimal)

 

Once MaxUserPort is at the maximum 65534, it's still possible for the rate of port use to exceed the rate at which they are returned to the pool. By default, a connection can't be reused for 2 times the Maximum Segment Lifetime (MSL). Effectively TcpTimedWaitDelay lets you set a value for the Time_Wait timeout manually. In most cases 30 seconds are an adequate value.

 

To take effect, the environment needs to be restarted.

 

 Suggestion B

There have been two bugs within versions before V10.5, which are addressed in 10.5 onward.

Please upgrade to 10.5 or later to address them.

 Next steps:

 

What to gather if these steps do not resolve the problem?

In case the issue occurs, gather the following data points from the Endpoint Server and provide them to Symantec DLP Support:

 

1. The  output of "netstat -a -b"

2. Process details of Endpoint Server processes

 

 a. Endpoint Server: What is the total resource (CPU, memory) consumption on the server.

 b. Endpoint Server: What is the resource consumption by our java processes on the endpoint server.

 c. Endpoint Server: What is the handle count for aggregator and file reader processes.

 d. Endpoint Server: Get an output of handles for these processes. Use handle.exe (http://technet.microsoft.com/en-us/sysinternals/bb896655.aspx ).

 

3. All Endpoint Server Logs.

4. Endpoint Server: Number of files in Vontu\Protect\Incidents.

5. While the Endpoint Server hangs, are other Detection Servers creating incidents?

6. Get a screen shot of the System Overview within the Enforce UI when logged in as Administrator.