DLP File Reader restarts excessively

Products

Data Loss Prevention Endpoint Prevent Data Loss Prevention Network Monitor Data Loss Prevention Network Prevent for Email Data Loss Prevention Enforce Data Loss Prevention Network Discover Data Loss Prevention Network Protect Data Loss Prevention Endpoint Discover Data Loss Prevention

Issue/Introduction

The File Reader for Data Loss Prevention (DLP) restarts excessively.

The Detection Server logs may show one or more of the following entries:

BoxMonitor log:

Restarted FileReader. FileReader process was restarted because it went down unexpectedly.

Process [FileReader] has not responded for 16 minute(s) 0 ms

Restarted FileReader. FileReader was restarted because it wasn't responding.

FileReader restarts excessively. Process FileReader has restarted x times during last xx minutes.

boxmonitor_operational log:

Process [FileReader] has not responded for 16 minute(s) 0 ms

Process FileReader was restarted

Process [FileReader] has not responded for 16 minute(s) 0

FileReader log:

Message chain #x didn't stop processing the message.

Failed to get data from the client within specified time., Exception thrown from : ServerShmChannelImpl.cpp(242) FileTypeIdentifierImpl.cpp 113

Content extraction for file [xxxxx.xxx] failed.

Recording Message processing failure due to ContentExtractionTimeoutException

Failed to process item: //<path to file xxxxx.xxx>. Content extraction timeout occurred.

Cause

File Reader restarts usually occur due to timeout issues, which can be caused by the following:

Connectivity issues with the Monitor Controller or Network
Low Java heap size
Poorly written RegEx rules
A bad email message, which can be caused by:
Incorrect header information
Foreign characters in the message that Symantec DLP is unable to process.

Resolution

If the File Reader restarts itself occasionally, this is normal behavior. However, if you are experiencing consistent File Reader restarts in your environment, there are a few things you can do to determine the cause:

File Reader may fail to start (and restart) if it can’t receive all the configuration information it needs.

Review File Reader logs to identify which File Reader subsystem isn’t starting.

Once identified that a particular subsystem isn’t receiving its configuration, review the MonitorController log to see if the corresponding subsystem has been initialized successfully.

One of the common failures is an inability to ignite cryptographic keys in the MonitorController because the ignition password on the disk got out of sync with the Administrator password in the database. In this case, the password issue must be fixed, and only after that should the MonitorController (SymantecDLPDetectionServerController) Service be restarted.

Java heap size increase

Review Monitor Controller properties file (SymantecDLPDetectionServer.conf) in the location:
Windows: <Drive>:\Program Files\Symantec\DataLossPrevention\DetectionServer\Services\SymantecDLPDetectionServer.conf
Linux: /opt/Symantec/DataLossPrevention/DetectionServer/Services/SymantecDLPDetectionServer.conf
Check the java heap size. If it is the default value of 128 and 256, increase the memory setting of the heap depending on the RAM available on the server.
1. Example:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory = 1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory = 2048

Policy configuration

FileReader restarts can occur due to a particular policy. An example is a regex in a particular policy exceeds given thresholds (such as maximum component time).

Review log files for an “intentionally restarting process” message which identifies the message chain component that is causing the restart.
If this component is “Detection,” the most likely cause is a poorly written regular expression. See Create efficient regular expressions to improve DLP performance

Too many exceptions in policies

Each new exception, while improving the ability to "short-circuit" detection, can multiply the size of the overall execution (detection) matrix - all of which is loaded into memory when File Reader starts.

While there is no exact limit on the total number of exception, a good rule of thumb is limit exception line count, leverage lists (Sender/Recipient) and avoid compound rules.

See High memory usage of DLP agent caused by compound exceptions.

MessageChain settings

Confirm that the MessageChain.CacheSize and MessageChain.NumChains match the number of CPU cores on the server.

For steps to edit these settings refer to the Advanced server settings help page.

Network Monitor

Check for "bad" messages in drop_pcap folder.
- Save the *.vpcap file that contains the message in question. This file can be used for testing without having to actually send the message again.
Check for locked *.vpcap files. Stop Packet Capture so that you do not get noise in the test, and start the File Reader process.
- If the *.vpcap file gets picked up, the inductor is working.
  - If the inductor is working, the problem may be in the Layer 7 Parser or the Content Extractor. Visually inspect the File Reader log for any exceptions, warnings or severe log messages.
- If the inductor is not working, the common problem is that a process has a lock on the files.
  - If there are no locked files, collect the File Reader log and contact Broadcom Support.

Note: Although Content Extractor can have problems processing various file formats, it will rarely cause File Reader restart.

Thread exceptions

Dying threads can cause File Reader to stop reporting heartbeats and eventually be restarted.

Review Look in BoxMonitor.log for exceptions. Each exception in that log file is an indicator of a serious problem (Java crash or another defect) and is a likely cause of a File Reader restart.

Additional help

If none of the above resolves your File Reader restart issues, contact Broadcom Support for further troubleshooting.