Entity missing from entity collection

Products

• Information Centric Analytics

Issue/Introduction

Contrary to expectations, one or more entities (i.e., persons, users, computer endpoints, IP addresses, applications, etc.) are not included in an entity collection. When viewed in the Information Centric Analytics (ICA) Analyzer, other entities are returned by the view definition. In comparing the dimensions and filters of the view to the attributes of the missing entity or entities, no differences can be found to explain their exclusion.

Environment

Release : 6.x

Cause

One or more conditions can be traced to this condition:

1. There is a discrepancy between an entity's attributes and the dimensions and filters used to define an entity collection
2. An entity either does not exist or its attributes are stale in the RiskFabric cube
3. An entity does not exist in the RiskFabric database

Assuming all entity attributes match the entity collection's definition, the most likely cause of this condition is the Analyzer is not returning current data.

The Analyzer displays data stored in the RiskFabric cube as measures and dimensions. The cube is fully processed once every 24 hours as part of the nightly RiskFabric Processing job, run by the SQL Server Agent. If enabled, the RiskFabric Intraday Processing job will run periodically throughout the day and process those measures and dimensions in the cube related to Data In Motion (DIM) incidents and their associated entities. Data from other data sources, including custom data sources created via the Integration Wizard (IW), will be staged during the day but only processed into the cube during the nightly job.

Similarly, entities from data sources like Active Directory (AD) are only processed into the RiskFabric database during the nightly job, as are user data pulled from custom data sources. Because data must first be processed into the core RiskFabric tables before they can be processed into the cube, ancillary staging jobs for custom data sources must run prior to the nightly job.

For example, if an entity collection's definition includes a filter to only include users who have the VIP value of '1' and that VIP flag is set by a query against a SQL data source, it's necessary for the data source query job to run in order to set that flag in both the RiskFabric database and cube during nightly processing. Likewise, if an entity collection's definition includes all users whose AD CountryAbbreviation is 'IE' but the Bay Dynamics AD Connector Job has not run recently, new AD users that match this definition will not be processed into the RiskFabric database and cube.

Resolution

To resolve this condition, first determine whether the cause is a mismatch between the entity's attributes and the entity collection's definition; a failure of a data source query job; a failure of the nightly job; or an issue upstream in the data source for either the entity or its custom attributes.

When comparing an entity's attributes to an entity collection's definition in the Analyzer, remember that a filter can be applied to each dimension regardless of whether the dimension has been added to the definition's Filters. Right-click each dimension listed under Rows and Columns and select Manager Filters to see a list of dimension members and their inclusions or exclusions.

To determine whether a data source query has failed, navigate in the ICA console to Admin > Integration > Job Status. Look for any jobs with a Last Run Outcome of Failed. Note the Name of the failed job and, using SQL Server Management Studio (SSMS), pass it to the following query to obtain more details about the cause of failure:

DECLARE    @JobName nvarchar(50) = '' /* Insert the failed job name between the single quotation marks on this line */
SELECT     j.[name],
           s.[step_name],
           h.[message] AS "Message"
FROM       msdb.dbo.sysjobs AS j
INNER JOIN msdb.dbo.sysjobsteps AS s 
        ON j.job_id = s.job_id
INNER JOIN msdb.dbo.sysjobhistory AS h WITH (NOLOCK)
        ON s.job_id = h.job_id AND
           s.step_id = h.step_id AND
           h.step_id <> 0
WHERE      s.last_run_outcome = 0 AND        
           j.[name] = @JobName
ORDER BY   h.instance_id DESC;

This query may also aid in identifying where data loading failures are occurring:

USE RiskFabric;
GO
SELECT   LogID,
         LogGroupID,
         LoginName,
         LogName,
         CAST(LogDescription AS nvarchar(255)) AS "LogDescription",
         StartDate,
         EndDate,
         RunMinutes,
         StatusFlag,
         SQLAgentJobName,
         StoredProcedureName,
         SourceTableName,
         DestinationTableName,
         DMLAction,
         RowsAffected,
         ErrorCode,
         ErrorDesc,
         ErrorSource,
         ErrorDate
FROM     Log_DataTransformation WITH (NOLOCK)
WHERE    StatusFlag = 'F'
ORDER BY 1;