Reprocessing records skipped during processing
search cancel

Reprocessing records skipped during processing


Article ID: 232341


Updated On:


Information Centric Analytics


When rows from an integration wizard (IW) data source are successfully processed into Information Centric Analytics' (ICA) staging tables, the data source query's watermark value is updated at the completion of the staging process. If processing of any row fails during this step, the watermark is not updated, the staging process halts, and the error message generated by the failure is written to the Log_DataTransformation table.

For processing failures that occur while updating ICA's core tables, however, processing will continue after a row failure under certain conditions because of dependencies in the processing of other core tables. Depending upon the number and frequency of failures, this can result in a significant discrepancy in event counts over time between the staging and core tables.


Release : 6.5.x

Component : Integration Wizard


To address a condition in which events are missing from ICA's core tables due to a previous core processing failure, the prescribed operation is as follows:

  1. Identify and fix the underlying cause of the processing failure
  2. Truncate the affected integration's staging table
  3. Set the data source query's watermark to a value preceding the failed row

This operation requires familiarity with the RiskFabric database schema. Please contact Broadcom support for assistance.