DIM incidents delayed or missing from ICA

Products

Information Centric Analytics

Issue/Introduction

Data In Motion (DIM) incidents from Symantec Data Loss Prevention (DLP) are delayed or do not appear in the Information Centric Analytics (ICA) console but are visible in the Enforce console.

Environment

Release : 6.x

Component : Symantec DLP Integration Pack

Cause

Symantec DLP incidents must first be reported to the Enforce server before they can be ingested by ICA. Once a DLP detection server reports an incident to Enforce, the incident will be ingested into ICA during the next intraday or nightly RiskFabric processing job run. Depending upon the frequency of these jobs' schedules and the gap between an incident's Occurred On and Reported On timestamps, there may be a significant delay between the date and time at which an incident occurred and its availability in the ICA database.

Consider the following examples:

A DLP endpoint agent installed on a laptop that was disconnected from the network detected a policy violation on April 1, 2021 at 23:50. This is the Occurred On date. The laptop was not reconnected to the network until 08:05 on April 2nd, at which point the agent reported the incident to Enforce by way of its detection server. This is the Reported On date. The RiskFabric Intraday Processing job was scheduled to run at the top of the hour every hour between 06:00 and 23:00 daily. Because the job at 08:00 had already started, the incident reported to Enforce at 08:05 wasn't processed into ICA until the 09:00 job ran. Cumulatively, this meant more than ten hours elapsed between when the incident occurred and when the incident was ingested into ICA, but only an hour or so elapsed between when the incident was first reported to Enforce and when it was ingested into ICA.
The same details as (1), but the RiskFabric Intraday Processing job is not enabled. Consequently, the incident that occurred on April 1st and was reported to Enforce on April 2nd is not ingested into ICA until the nightly job runs at 00:15 on April 3rd, resulting in a delay of ≈48 hours between the time of the incident's occurrence and the incident's appearance in the ICA console.

Alternatively, you confirm an incident has been ingested into ICA within the timeframe expected but it does not appear in certain dashboards. These dashboards (or their underlying Analyzer views or searches) include a filter on incident status. When an incident is created in DLP, the IncidentPersister service creates a record for the incident in the DLP database. This is a multi-part process that may take several hours to complete, under certain circumstances. If ICA's staging queries run prior to all values being populated in the DLP database - like the incident status - the incident record will be imported into ICA without those values present. Subsequent processing jobs will import any values that are later added to the incident record but, initially, a new incident could be imported without yet being assigned the status value of 'New'. Consequently, if a dashboard in ICA is configured to only display incidents with a status of 'New', these new incidents will not be displayed until the persister service has recorded that status on a new incident.

Resolution

If you suspect there is an abnormal delay before an incident is available in ICA, correlate the following pieces of information between DLP and ICA:

Message Date
Detection Date
Creation Date
Action Date
SQL Server Agent job history for the RiskFabric processing jobs
Creation Date in the RiskFabric relational database

The following queries may be of help in troubleshooting the incident creation and ingestion process:

Incident Creation and Persistence History

Execute in SQL*plus or Oracle SQL Developer against the DLP Protect relational database, replacing <ID> with the ID of the incident under investigation:

SELECT
    i.INCIDENTID,
    i.MESSAGEDATE,
    i.DETECTIONDATE,
    i.CREATIONDATE,
    a.ACTIONDATE
FROM
    PROTECT.INCIDENT i
LEFT JOIN
    PROTECT.INCIDENTACTION a ON i.INCIDENTID = a.INCIDENTID
WHERE
    a.ACTIONDETAIL = 'incident.status.New' AND
    i.INCIDENTID = <ID>;

SQL Server Agent job history

Execute in SQL Server Management Studio (SSMS) against the Database Engine hosting the RiskFabric relational database, replacing <ID> with the ID of the incident under investigation:

DECLARE     @id int,
            @date datetime;

SET         @id = <ID>;
SET         @date = (SELECT IncidentDate FROM RiskFabric.dbo.LDW_DIMIncidents WITH (NOLOCK) WHERE SourceIncidentID = @id);

SELECT      h.instance_id,
            j.[name] AS "Job",
            s.step_id AS "StepID",
            s.step_name AS "StepName",
            h.[message] AS "Message",
            msdb.dbo.agent_datetime(run_date, run_time) AS "RunDateTime",
            ((h.run_duration/10000*3600 + (h.run_duration/100)%100*60 + h.run_duration%100 + 31 ) / 60) AS "Minutes"
FROM        msdb.dbo.sysjobs AS j
INNER JOIN  msdb.dbo.sysjobsteps AS s 
        ON  j.job_id = s.job_id
INNER JOIN  msdb.dbo.sysjobhistory AS h WITH (NOLOCK)
        ON  s.job_id = h.job_id AND
            s.step_id = h.step_id AND
            h.step_id <> 0
WHERE       (
                j.[name] = 'RiskFabric Processing' OR
                j.[name] = 'RiskFabric Intraday Processing'
            ) AND
            CAST(msdb.dbo.agent_datetime(run_date, run_time) AS date) >= CAST(@date AS date)
ORDER BY    h.instance_id DESC;

ICA Incident Ingestion History

Execute in SSMS against the RiskFabric relational database, replacing <ID> with the ID of the incident under investigation:

SELECT DIMIncidentID,
       SourceIncidentID,
       RFCreatedDate AS "IngestionDate"
FROM   RiskFabric.dbo.LDW_DIMIncidents WITH (NOLOCK)
WHERE  SourceIncidentID = <ID>;

Additional Information

Refer to the following KB articles for more information: