Incidents are not being processed - queueing up on the Manager

Products

• Data Loss Prevention Enforce

Issue/Introduction

The Monitors are processing the incidents but the number of incidents is growing on the Enforce server.

Resolution

There can be multiple reasons why this issue occurs. Below are a few things you can do to attempt to resolve this issue:

1. If incidents are queuing in the incidents directory of Enforce (<installDrive>:\ProgramData\Symantec\DataLossPrevention\ServerPlatformCommon\<version>\incidents):

1. Check the IncidentPersister.log If the log shows errors, see if you can interpret the error.
2. It could be a connectivity issue with the database. Check for any exceptions or Oracle errors.
3. Make sure the services are running.
   1. To check if the Symantec DLP Enforce services are running on Linux:  service --status-all
   2. On Windows:  Go to Control Panel -> Administrative Tools -> Services
4. Make sure that the database is available.
5. Make sure the permissions on the incidents directory are valid for the Service Account (typically SymantecDLP or PROTECT user).
6. Restart the Symantec DLP Incident Persister Service.

 

2. The Manager is failing to run the response rules against the incidents. 

1. If the incidents directory is relatively empty, the incidents may be queueing in the database.
2. Look in the tomcat and SymantecDLPManager logs to see if there are errors indicating the failure to process incidents.
3. The errors could be caused by the attribute lookup failing, or the failure to send out notification emails.
4. You can check how many incidents are queued in the database by running:
   select count(*) from incident where incidentstatusid is null;
   If this count is growing, then the response rules are either not running, or running slowly.

 

3. The Oracle DB could be out of temp space (99.9 %). If this occurs, the incidents will queue up because they are not being written back to the database. Adding more tablespace will resolve it. To add more tablespace with Oracle, see the following article:

Oracle tablespace (LOB_TABLESPACE, USERS, etc.) for DLP is full

4. Further more related issues can be checked using below articles:

Incidents no longer generate after server upgrade

Java heap space' occurs when persisting large incidents from an Endpoint Data at Rest discover scan