DLP Cloud Detector incident queue stays at 1000 in Enforce console

Products

• Data Loss Prevention Cloud Detection Service for REST
• Data Loss Prevention Cloud Detection Service for ICAP
• Data Loss Prevention Cloud Detection Service
• Data Loss Prevention Cloud Package

Issue/Introduction

It has been noted in the Enforce management console that the Incident Queue for your Cloud Detector is stuck at 1000 and doesn't seem to be changing.

• You aren't seeing a high volume of incidents from this Detector, and don't believe there are DLP policies that would warrant the backlog that you are currently noticing.
• You are wanting to know if there is any kind of backend issue with the Detector. 

Environment

Release : 15.x

Component : Cloud Detection Server

Cause

The DLP Cloud Service is a hosted service which integrates with the Enforce console - itself a managed on-prem installation.

In contrast to on-prem Detection Servers, the Incident Queue for a Cloud Detector can report up to a maximum of 1000 incidents.

In this state, incidents are queued at the Cloud Service Gateway, and will remain there until successfully shipped to Enforce.

The backlog as such is related to how many incidents have been created on the Detector, but is also dependent upon the Enforce server's internet-connection speed to the Cloud Service Gateway, and on the resource availability for the Enforce DLP services processing incidents.

Resolution

Summary of issue:

• The Incident Queue will remain at 1000 as long as there are more than 1000 incidents remaining on the Detector.
• This cap applies to all Cloud Service Detectors enrolled in a single Enforce Server. Thus, if you have 2 Detectors creating incidents, the "incident queue" for each will show <1000. If the total count of queued incidents of both adds up to 1000 - then there are more incidents awaiting shipping at the Cloud Service Gateway.
• That number can only go down once the rate of new incidents drops below the rate at which incidents can be shipped to Enforce.

Steps to take:

• To verify the number of backlogged incidents which were recently generated, versus those which have recently "shipped" to Enforce, you can open a case with Technical Support.
• To reduce this issue from recurring, policies creating high volumes of incidents may need to be tuned to eliminate False Positives, as well as Duplicate incidents (multiple incidents triggered by a single event such as one file upload or email message).
• You can also increase the memory allocation to specific Enforce services, such as the SymantecDLPDetectionServerController and the SymantecDLPIncidentPersister. For details on that please see this article: Monitor Controller performance issues after adding new Detection Servers (broadcom.com).

Additional Information

Also, for customers on DLP 15.8 MP2 or earlier see the following KB: Incidents from the DLP Cloud Service are stuck in queue (broadcom.com).