Monitor Controller performance issues after adding new Detection Servers

Products

Data Loss Prevention Enforce Data Loss Prevention Network Discover Data Loss Prevention Cloud Service for Email Data Loss Prevention Cloud Detection Service Data Loss Prevention Cloud Package Data Loss Prevention

Issue/Introduction

The SymantecDLPDetectionServerController service* on Enforce may use more memory in later versions of DLP than in earlier ones. The memory usage is dependent on a number of factors:

The number of Detection Servers
How many EDMs or other two-tier indices are being utilized
Whether there are any Network Discover scans involved, especially those utilizing Incremental scanning (which sends updates of an incremental scan index to all servers during scans)
Cloud Detection Servers (for both Email and Application Detection) also seem to require more memory usage by the MonitorController, for the distribution of their profiles

In general, large deployments may run into a need for more memory for the Detection Server Controller.

*Note - in previous versions of DLP, this service was known as the Monitor Controller.

Perhaps you are seeing frequent "RSODs" (a red bar error appearing in Enforce Server), with the following detail:

Failed to contact the server controller. Make sure Symantec DLP Detection Server Controller service is running.

In many cases, the addition of a second or third Cloud Detector puts these detection servers into a "Disconnected" state - usually resolved by recycling the Detection Server Controller service.

Environment

All supported versions of DLP.

Cause

Based on load and conditions listed, the DetectionServerController (aka MonitorController) would benefit from tuning for better performance.

Resolution

To increase the memory for the Monitor Controller, modify your installation as per the following:

In supported versions, update SymantecDLPDetectionServerController.conf file, located by default in this DLP directory:

■ Windows:

\Program Files\Symantec\DataLossPrevention\EnforceServer\Services

■ Linux:

/opt/Symantec/DataLossPrevention/EnforceServer/Services

Below are the current defaults for all supported versions (15.8 and 16.0 as of 2023):

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

For specific recommendations, see the following Help Center topics:

The recommendations on that page cover the "maxmemory" setting. In general, the "initmemory" should be 50% of that.

Per the examples, for a medium installation of DLP, the following settings are recommended:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=8192

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=16384

For suggestions about which size installation you have, see The Effect of Scale on System Requirements (broadcom.com).

After making the above changes, be sure to restart the SymantecDLPDetectionServerController process or service.

Note: Be sure to confirm the amount of memory installed on the server before modifying beyond above recommendations.

Do not increase the memory beyond 31GB.
At 32GB you lose memory compression and it becomes counter-productive.
In most circumstances there are better ways to handle out of memory errors than increasing the memory beyond 31GB.

Additional Information

Sometimes the following errors are present for the performance issues given in this KB, but the absence of these errors does not mean performance issues do not exist.

MonitorController0.log:

com.vontu.command.loader.ModelEventHandler$ReloadCommandInstructionsTask run
SEVERE: Could not reload command instructions
java.lang.OutOfMemoryError: GC overhead limit exceeded

SymantecDLPDetectionServerController.log:

Exception in thread "Incidents_application_updaterWorker_1" java.lang.OutOfMemoryError: Java heap space