Endpoint incidents queued after upgrading to DLP 15.8 or later.

Products

• Data Loss Prevention Endpoint Prevent

Issue/Introduction

After upgrading to 15.8 incidents are not being reported by endpoint agents, or are not being reported in a timely manner.

In Aggregator#.Log you may find the following.

INFO: Current matrix cache size (in bytes) is: <Some number greater than 1000000000>. and memory limit setting (in bytes) is: 1000000000. Need to do cache eviction.

This suggests that we need to increase the size of this cache and the amount of memory available on the server for endpoint communication.

 

Cause

DLP 15.8 and later has the ability to send individual policies to the agent, prior to 15.8, any policy update required sending the entire policy set to the agent.

This is done by the agents sending policy data to the endpoint servers, so when policies change the endpoint server can identify which policies need to be sent to the agent.

However this, depending on the environment, can use a lot of memory on the endpoint server.  The biggest factors are the following:

Amount and complexity of policies

Total number of agents.

Load Balancer settings

 

Resolution

1. Identify the size of the execution matrix.

To do this, set an agent's logging to FINEST and restart the agent.

Search for the last line containing "Adding Row:" in the edpa_ext0.log file. This will give you the number of rows in the execution matrix for this agent.

2. Identify how many agents, in total, each server is configured to receive.

3. Multiply the number of rows in the exeuction matrix * 16 * number of agents a server may 'see' over time.

This will give you the total number of bytes needed to store policy deltas in cache on the endpoint server.

4. Modify The following line in Aggregator.properties on all considered Endpoint servers to match the calculated storage requirement.

# ExecutionMatrix LRU cache limit for Aggregator
matrixCacheLimitInBytes = 1000000000

5. Increase BoxMonitor.EndpointServerMemory by the same amount of memory that was increased in Aggregator.properties.

6. Recycle the services on the Endpoint server.