Monitor Controller performance issues after adding new Detection Servers

Products

Data Loss Prevention Enforce Data Loss Prevention Network Discover Data Loss Prevention Cloud Service for Email Data Loss Prevention Cloud Detection Service Data Loss Prevention Cloud Package Data Loss Prevention

Issue/Introduction

The SymantecDLPDetectionServerController service* on Enforce may use more memory in later versions of DLP than in earlier ones. The memory usage is dependent on a number of factors:

The number of Detection Servers
How many EDMs or other two-tier indices are being utilized
Whether there are any Network Discover scans involved, especially those utilizing Incremental scanning (which sends updates of an incremental scan index to all servers during scans)
Cloud Detection Servers (for both Email and Application Detection) also seem to require more memory usage by the MonitorController, for the distribution of their profiles

In general, large deployments may run into a need for more memory for the Monitor Controller.

*Note - prior to 15.1, this service was known as the Monitor Controller.

Perhaps you are seeing frequent "RSODs" (a red bar error appearing in Enforce Server), with the following detail:

Error: Failed to contact the server controller. Make sure Vontu Monitor Controller service is running.

Either of the following errors may also be present in the logs - but the service doesn't always encounter this exception.

MonitorController0.log:

com.vontu.command.loader.ModelEventHandler$ReloadCommandInstructionsTask run
SEVERE: Could not reload command instructions
java.lang.OutOfMemoryError: GC overhead limit exceeded

SymantecDLPDetectionServerController.log:

Exception in thread "Incidents_application_updaterWorker_1" java.lang.OutOfMemoryError: Java heap space

Cause

Based on load and conditions listed, the DetectionServerController (aka MonitorController) would benefit from tuning for better performance.

Environment

All supported versions of DLP.

Resolution

To increase the memory for the Monitor Controller, modify your installation as per the following:

In versions 15.5 and later, update SymantecDLPDetectionServerController.conf file, located by default in this DLP directory:

■ Windows:

\Program Files\Symantec\DataLossPrevention\EnforceServer\Services

■ Linux:

/opt/Symantec/DataLossPrevention/EnforceServer/Services

Below are the default settings, in version 15.5 and earlier:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=128

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=1024

Below are the recommended settings going forward, for all versions:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

After making the above changes, be sure to restart the SymantecDLPDetectionServerController process or service.

Note: For better performance, or in very large enterprises, these settings can be increased further, even to 8 and 16 GB, respectively. Be sure to confirm the amount of memory installed on the server before modifying beyond above recommendations. A good rule of thumb is to set the maxmemory to <= 25% of RAM on the box.