Error 'java.lang.OutOfMemoryError: Java heap space' occurs when persisting large incidents from an Endpoint Data at Rest discover scan

Products

Data Loss Prevention Endpoint Suite Data Loss Prevention Endpoint Discover Data Loss Prevention Enforce

Issue/Introduction

Error 'java.lang.OutOfMemoryError: Java heap space' occurs when handling large incidents from an Endpoint Data at Rest discover scan. You may also see another variant of this error - 'java.lang.OutOfMemory gc overhead limit exceeded'.

Observations

Note - Depending on each service's current java heap settings and the actual size of the .IDC files being processed, you may not see OutOfMemory errors in all of the services listed below:

You observe 'java.lang.OutOfMemoryError: Java heap space' errors in the IncidentPersister logs while persisting the large incidents.
- There are large .IDC files in the Enforce incidents folder that don't process.
- You may also observe large .IDC files remaining in the incidents folder on the Endpoint Detection Server once the default of 15k incidents held at the Enforce server has been reached (Enforce, IncidentPersister.properties file, persister.max_incidents_in_folder parameters).

You observe the following OutOfMemory error in the Aggregator log on an EPS server that is receiving many large incidents from endpoint agents.

com.symantec.dlp.communications.common.activitylogging.JavaLoggerImpl log
SEVERE: Stack array is empty. The following exception does not have a proper stack trace.
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
 at com.symantec.dlp.communications.common.activitylogging.ConnectionLogger.getThrottler(ConnectionLogger.java:547)

You observe 'java.lang.OutOfMemoryError: Java heap space' errors in the MonitorController(n).logs on Enforce while receiving the .IDC files from the Endpoint Detection Server.

The Oracle server encounters the following error while IncidentPersister attempts to persist very large .IDC files:

com.vontu.util.jdbc.DatabaseRuntimeException: java.sql.SQLRecoverableException: ORA-01034: ORACLE not available
ORA-27102: out of memory

You can observe these same symptoms even when the IncidentHandler.MAX_INCIDENT_FILE_SIZE.int is at the default of 30MB or even if it has been lowered.

Environment

DLP 15.x

Cause

The incidents being persisted exceed the max Java heap size allocated to the affected services.

Resolution

If the files that trigger the incidents during the scan do not need to be retained

Remove the Limit Incident Data Retention Response Rules that have "Retain Original Message" enabled from the policies that are being used in the discover scans.

If the files that trigger the incidents need to be retained

Retention 1

In order to accommodate very large incident files, you will have to increase the Java max heap sizes for the affected DLP services, and the Oracle database server memory_target to a point that the large .IDC files can be persisted. How high you have to go on each of these is dependent on the size of the .IDC files being received, and the rate at which they are being received by the Endpoint Detection Server and Detection Server Controller Service (MonitorController). For IncidentPersister, both the size, and number of threads being utilized to persist incidents (Enforce, IncidentPersister.properties, persister.threadPoolSize parameter) affects how high the max heap size needs to be:

- Symantec DLP Detection Controller service on Enforce:
  1. Increase the wrapper.java.maxmemory value in:
    Windows: C:\Program Files\Symantec\DataLossPrevention\EnforceServer\Services\SymantecDLPDetectionServerController.conf or
    Linux: /opt/Symantec/DataLossPrevention/EnforceServer/Services/SymantecDLPDetectionServerController.conf
  2. Restart the Symantec DLP Detection Server Controller service.
- Symantec DLP Incident Persister service on Enforce:
  1. Increase the wrapper.java.maxmemory value in:
    Windows: C:\Program Files\Symantec\DataLossPrevention\EnforceServer\Services\SymantecDLPIncidentPersister.conf or
    Linux: /opt/Symantec/DataLossPrevention/EnforceServer/Services/SymantecDLPIncidentPersister.conf
  2. Restart the Symantec Incident Persister service.
- If this is an Endpoint Discover Scan - Aggregator on the affected Endpoint Detection Server:
  1. Increase the -Xmx value for BoxMonitor.EndpointServerMemory in the EPS' Advanced Settings.
  2. Restart the Symantec DLP Detection Server service on the Endpoint Detection Server.
- Oracle database service:
  1. Review logs for 'ORA-27102: out of memory' errors. If you see these errors while processing the large incidents, work with your DBA to increase the Oracle database server's memory_target.
  2. Review the LOB_TABLESPACE utilization. If very large files are retained along with incidents frequently (persisted in the MessageComponentLob.UncrackedComponent field), this will cause LOB_TABLESPACE usage to increase very quickly and you may need to add more datafiles, and frequently.
    1. Consider externalizing BLOBs to file system storage, instead of database storage:
      1. In Protect.properties on Enforce
        
        Configure the com.symantec.dlp.incident.blob.externalization.dir value
        
        Set com.symantec.dlp.incident.blob.externalize = true
        
        Restart the Symantec DLP Incident Persister service.

Retention 2

Set an upper limit of 150MB on the file sizes to scan. In the Target Filters tab, set a value in the "Ignore Greater than" field, as shown below:
Also consider adding more file types or paths to the Exclude Filters field shown above.

Additional Information

Endpoint Discover Notes

Endpoint discover scans do not observe the Agent Advanced Setting "IncidentHandler.MAX_INCIDENT_FILE_SIZE.int" (default of 30MB). This behavior, combined with having a "Limit Incident Data Retention" Response Rule with the option of "Supported Endpoint Channels -> Retain Original Message" enabled on the policies that are being used in the EDAR discover scan at the endpoints, is what can cause the large incidents to be created.
Unlike the Data In Motion (DIM) behavior of the Advanced Agent setting "Detection.MAX_FILTER_FILE_SIZE.int" which extracts and scans up to the amount specified from each file or subfile, this EDAR setting of "Ignore Greater than" causes the scan to completely ignore files that exceed this threshold.
As of DLP 15.7 the maximum supported content inspection size for Endpoint Detection is 150MB as documented at: Guidelines for tuning Symantec Data Loss Prevention to scan large files
- If utilizing a response rule to retain endpoint messages (files) it's important that the Endpoint Discover Target have a limit in the "Ignore Greater than" field of no more than 2000MB, because incidents with retained files over 2GB cannot be persisted due the limit on a Java array capacity not exceeding Integer.MAX_VALUE (2,147,483,647).