Reprocessing records skipped during processing

Products

• Information Centric Analytics

Issue/Introduction

When rows from an integration wizard (IW) data source are successfully processed into Information Centric Analytics' (ICA) staging tables, the data source query's watermark value is updated at the completion of the staging process. If processing of any row fails during this step, the watermark is not updated, the staging process halts, and the error message generated by the failure is written to the Log_DataTransformation table.

For processing failures that occur while updating ICA's core tables, however, processing will continue after a row failure under certain conditions because of dependencies in the processing of other core tables. Depending upon the number and frequency of failures, this can result in a significant discrepancy in event counts over time between the staging and core tables.

Environment

Release : 6.5.x

Component : Integration Wizard

Resolution

To address a condition in which events are missing from ICA's core tables due to a previous core processing failure, the prescribed operation is as follows:

1. Identify and fix the underlying cause of the processing failure
2. Truncate the affected integration's staging table
3. Set the data source query's watermark to a value preceding the failed row

This operation requires familiarity with the RiskFabric database schema. Please contact Broadcom support for assistance.