DLP Network prevent for Email Detection Server triggered incident before EDM profiles were loaded.

Products

• Data Loss Prevention Enterprise Suite
• Data Loss Prevention Network Prevent for Email

Issue/Introduction

Network Prevent for Email detection server scanned email messages after detection service restart while EDM profiles were still loading.

The detection server encountered a restart due to a message chain timeout.

At this time it an incident was generated where an EDM profile used in a policy exception did not load before the message was scanned and in the detection_operational log we can see the following timeline: 

•    Detection service is restarting intentionally after message chain timeout

25/Jun/24:09:24:18:451+0800 [INFO] (DETECTION.4) Detection is shutting down
25/Jun/24:09:24:43:996+0800 [INFO] (DETECTION.1) Detection is starting

•    Policies and EDM profiles are loading for approx. 5 mins after the service restart

25/Jun/24:09:25:30:751+0800 [INFO] (DETECTION.300) Profile [106,462:49][...] loaded successfully
<<<<<<<<<<<<<<<<<<<<<<<<<<<output omitted>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
25/Jun/24:09:26:37:208+0800 [INFO] (DETECTION.300) Profile [104,892:38][EDM_Profile_1 - used in detection rule] loaded successfully
<<<<<<<<<<<<<<<<<<<<<<<<<<<output omitted>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
25/Jun/24:09:28:29:574+0800 [INFO] (DETECTION.300) Profile [104,891:38][EDM_Profile_2 - used in exception rule] loaded successfully
<<<<<<<<<<<<<<<<<<<<<<<<<<<output omitted>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
25/Jun/24:09:30:01:422+0800 [INFO] (DETECTION.300) Profile [106,806:1,353][...] loaded successfully
25/Jun/24:09:30:02:460+0800 [INFO] (DETECTION.300) Profile [106,872:187][...] loaded successfully

•    Email is inspected at approx. 9:27am and incident is generated because the EDM_Profile_2 - used in exception is not loaded

Cause

The EDM used in an exception was not enforced at the time of the message inspection and the incident was generated.

Resolution

Policy loading and index loading are asynchronous, to prevent large detection 'gaps' - index loads cannot block detection and vice versa.

Typically this leads to policies being loaded first to allow detection to start quickly and prevent backups in message queueing (causing other problems).

For an index based leaf condition, the condition is skipped until the applicable index is loaded. Index updates are also asynchronous, as an aside once an index for a condition is

loaded updates don't cause it to be skipped it's only when no index is loaded usually, but not always around start up time.

In this case the condition was an exception and the other rules in the policy which didn't require indexes that had not been loaded were run - which generated a match.

It is expected behaviour as per product design as confirmed by Engineering. The policies fail open (meaning they skip conditions which aren't loaded).

Additional Information

Two Feature Requests are currently open to give more control on the behavior (2024/09/26): 

ISFR-2833 - [Detection] Customer should be able to control when to fail open or fail close. 

ISFR-3405 - Provide ability to disable detection until all policies and profiles are loaded.