Monitor Controller performance issues after adding new Detection Servers

Products

• Data Loss Prevention Enforce
• Data Loss Prevention Network Discover
• Data Loss Prevention Cloud Service for Email
• Data Loss Prevention Cloud Detection Service
• Data Loss Prevention Cloud Package

Issue/Introduction

The Monitor Controller service* on Enforce may use more memory in later versions of DLP than in earlier ones.  The memory usage is dependent on a number of factors:

• The number of Detection Servers
• How many EDMs or other two-tier indices are being utilized
• Whether there are any Network Discover scans involved, especially those utilizing Incremental scanning (which sends updates of an incremental scan index to all servers during scans)
• Cloud Detection Servers (for both Email and Application Detection) also seem to require more memory usage by the MonitorController, for the distribution of their profiles

In general, large deployments may run into a need for more memory for the Monitor Controller.

*Note - in 15.1 and above, this service was renamed to SymantecDLPDetectionServerController.

Perhaps you are seeing frequent "RSODs" (a red bar error appearing in Enforce console), with the following detail:

Failed to contact the server controller. Make sure Vontu Monitor Controller service is running.

 

Either of the following errors may also be present in the logs - but the service doesn't always encounter this exception.

MonitorController0.log:

com.vontu.command.loader.ModelEventHandler$ReloadCommandInstructionsTask run
SEVERE: Could not reload command instructions
java.lang.OutOfMemoryError: GC overhead limit exceeded

 

SymantecDLPDetectionServerController.log (aka VontuMonitorController.log):

Exception in thread "Incidents_application_updaterWorker_1" java.lang.OutOfMemoryError: Java heap space

Cause

Based on load and conditions listed, the DetectionServerController (aka MonitorController) would benefit from tuning for better performance.

Environment

All supported versions of DLP.

 

NOTE (for cloud related memory issues): 

In 14.6.x, cloud related Java heap errors are present in VontuMonitorController logs and NOT MonitorController logs.

In 15.x, cloud related Java heap errors are present MonitorController logs and NOT VontuMonitorController logs.
 

Resolution

To increase the memory for the Monitor Controller, modify your installation as per the following:

In versions 15.1 and later, update SymantecDLPDetectionServerController.conf, located in this DLP directory:

■ Windows:
\Program Files\Symantec\DataLossPrevention\EnforceServer\Services

■ Linux:

/opt/Symantec/DataLossPrevention/EnforceServer/Services

 

In versions 15.0 and earlier, update VontuMonitorController.conf, located in this DLP directory:

■ Windows:

\SymantecDLP\Protect\config

■ Linux:
/opt/SymantecDLP/Protect/config

 

 

Below are the version 14.x and 15.0 defaults:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=128

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=1024

 

 

Below are the recommended settings going forward, for all versions:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

 

After making the above changes, be sure to restart the SymantecDLPDetectionServerController or the VontuMonitorController process or service.

Note: For better performance, or in very large enterprises, these settings can be increased further, even to 8 and 16 GB, respectively. Be sure to confirm the amount of memory installed on the server before modifying beyond above recommendations. A good rule of thumb is to set the maxmemory to <= 25% of RAM on the box.