Monitor Controller performance issues after adding new Detection Servers

Products

Data Loss Prevention Enforce Data Loss Prevention Network Discover Data Loss Prevention Cloud Service for Email Data Loss Prevention Cloud Detection Service Data Loss Prevention Cloud Package Data Loss Prevention

Issue/Introduction

The SymantecDLPDetectionServerController service* on Enforce may use more memory in later versions of DLP than in earlier ones. The memory usage is dependent on a number of factors:

The number of Detection Servers
How many EDMs or other two-tier indices are being utilized
Whether there are any Network Discover scans involved, especially those utilizing Incremental scanning (which sends updates of an incremental scan index to all servers during scans)
Cloud Detection Servers (for both Email and Application Detection) also seem to require more memory usage by the MonitorController, for the distribution of their profiles

In general, large deployments may run into a need for more memory for the Monitor Controller.

*Note - prior to 15.1, this service was known as the Monitor Controller.

Perhaps you are seeing frequent "RSODs" (a red bar error appearing in Enforce Server), with the following detail:

Error: Failed to contact the server controller. Make sure Vontu Monitor Controller service is running.

In many cases, the addition of a second or third Cloud Detector puts these detection servers into a "Disconnected" state - usually resolved by recycling the DS Controller service.

Environment

All supported versions of DLP.

Cause

Based on load and conditions listed, the DetectionServerController (aka MonitorController) would benefit from tuning for better performance.

Resolution

To increase the memory for the Monitor Controller, modify your installation as per the following:

In supported versions, update SymantecDLPDetectionServerController.conf file, located by default in this DLP directory:

■ Windows:

\Program Files\Symantec\DataLossPrevention\EnforceServer\Services

■ Linux:

/opt/Symantec/DataLossPrevention/EnforceServer/Services

Below are the current defaults for all supported versions (15.8 and 16.0 as of 2023):

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

For customers of the DLP Cloud Services, it is recommended to increase this default further.

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=2048

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=4096

After making the above changes, be sure to restart the SymantecDLPDetectionServerController process or service.

Note: For better performance, or in very large enterprises, these settings can be increased further, even to 8 and 16 GB, respectively. Be sure to confirm the amount of memory installed on the server before modifying beyond above recommendations. A good rule of thumb is to set the maxmemory to <= 25% of RAM on the box.

Additional Information

Sometimes the following errors are present for the performance issues given in this KB, but the absence of these errors does not mean performance issues do not exist.

MonitorController0.log:

com.vontu.command.loader.ModelEventHandler$ReloadCommandInstructionsTask run
SEVERE: Could not reload command instructions
java.lang.OutOfMemoryError: GC overhead limit exceeded

SymantecDLPDetectionServerController.log:

Exception in thread "Incidents_application_updaterWorker_1" java.lang.OutOfMemoryError: Java heap space