Cloud Detector Incident Queue maxed out at 1000 - no persisted incidents for several days

book

Article ID: 170288

calendar_today

Updated On:

Products

Data Loss Prevention Cloud Prevent for Microsoft Office 365 Data Loss Prevention Cloud Service for Email Data Loss Prevention Cloud Detection Service Data Loss Prevention Cloud Service for Email with Cloud Console Data Loss Prevention Data Loss Prevention Enforce

Issue/Introduction

You have issues with your Cloud Detector(s) and have the following issues:

  • Several days with no new incidents from a Cloud Detector.
  • Incident queue, as it relates to the Cloud Detector has maxed out at 1000.
  • No incidents reported under 'Last 24 hours'.
     

 

Protect\logs\debug\VontuMonitorController.log (\SymantecDetectionServerController.log in 15.1 and newer)
Exception in thread "Incidents_application_updaterWorker_1" java.lang.OutOfMemoryError: Java heap space

Cause

Incidents are queued on the Cloud Detector(s) and cannot be processed by Enforce after hitting 1000 queued incidents because of the Java memory heap filling up.

 

Note: If you have more than 1 Cloud Detector, the total number of queued incidents between all of your Cloud Detectors combined will be 1000 queued incidents.

Resolution


Increase Java Heap memory in the VontuMonitorController.conf (SymantecDetectionServerController.conf in 15.1 and newer) file located here: Protect\config\VontuMonitorController.conf

In versions 15.1 and later, update SymantecDLPDetectionServerController.conf, located in this DLP directory:

■ Windows:
\Program Files\Symantec\DataLossPrevention\EnforceServer\Services
■ Linux:
/opt/Symantec/DataLossPrevention/EnforceServer/Services



NOTE: Before increasing JVM memory, ensure the system has ample free memory or usable standby memory.

Example: Original Values
# Initial Java Heap Size (in MB)
wrapper.java.initmemory = 128
wrapper.java.maxmemory = 2048

Example: Sample Values
# Initial Java Heap Size (in MB)
wrapper.java.initmemory = 1024
wrapper.java.maxmemory = 4096

After increasing the Java Heap init and max values, restart the VontuMonitorController or SymantecDetectionServerController service and wait a few minutes. You should see the queue numbers start dropping and the number of processed incidents go up as the incidents are sent to the Enforce Server to be written to the database.

Depending on the amount of time that the queue was backed up, it may take hours or days to completely parse through the backed up incident queue.