Enforce server incident queue backlogged

Products

• Data Loss Prevention Enforce
• Data Loss Prevention

Issue/Introduction

The Symantec Data Loss Prevention (DLP) Enforce server incident queue is backlogged and the incident count remains the same. Additionally, the detection servers incident queues continue growing. Recycling the Incident Persister service on the Enforce server allows some incidents to get processed, but the queue eventually comes to a stop. Checking the incident persister log shows an out of memory error within a minute after initializing:

INFO   | jvm 1    | 2021/05/26 14:12:46 | WrapperManager: Initializing...
INFO   | jvm 1    | 2021/05/26 14:13:40 | Exception in thread "pool-7-thread-3" java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at java.util.Arrays.copyOf(Arrays.java:3236)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.util.stream.StreamReader.read(StreamReader.java:78)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.util.stream.StreamReader.read(StreamReader.java:113)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.util.stream.StreamReader.read(StreamReader.java:102)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.util.stream.StreamReader.read(StreamReader.java:92)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.incidenthandler.message.v14.SerializableMessage.extractOriginalContent(SerializableMessage.java:223)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.incidenthandler.message.v14.SerializableMessage.extractComponents(SerializableMessage.java:239)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.incidenthandler.message.v14.SerializableNetworkMessage.createCopyExtractingOriginalContent(SerializableNetworkMessage.java:88)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.incidenthandler.message.persist.reconcile.IncidentReconciliator.reconcile(IncidentReconciliator.java:104)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at com.vontu.incidenthandler.message.persist.IncidentPersistingThread.run(IncidentPersistingThread.java:140)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2021/05/26 14:13:40 |  at java.lang.Thread.run(Thread.java:748)

Environment

Component : Enforce / Incident Persister

Cause

Incident persister requires additional memory to process large incidents or more efficiently process a large number of incidents, 100,000+ per day, for example.

Resolution

Increase the maximum java heap size in the SymantecDLPIncidentPersister.conf file.

Symantec DLP Incident Persister service on Enforce:

1. 1. Increase the wrapper.java.maxmemory value in:
      Windows: C:\Program Files\Symantec\DataLossPrevention\EnforceServer\Services\SymantecDLPIncidentPersister.conf or
      Linux: /opt/Symantec/DataLossPrevention/EnforceServer/Services/SymantecDLPIncidentPersister.conf
   2. Restart the Symantec Incident Persister service.

Check the amount of available system memory, then increase the Java Heap Size until the incident persister service can run without hitting the out of memory condition.

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory = 4096


Do not increase the memory beyond 31GB.
At 32GB you lose memory compression and it becomes counter-productive.
In most circumstances there are better ways to handle out of memory errors than increasing the memory beyond 31GB.