DWH Full load is missing data randomly

book

Article ID: 207222

calendar_today

Updated On:

Products

Clarity PPM On Premise Clarity PPM SaaS STARTER PACK-CLARITY PPM

Issue/Introduction

After the Load Data Warehouse runs, we found that some data is missing randomly.  We noticed it happens more after a performance issue in our system that caused the Full load to process for several hours at a time. We see missing data in the DWH tables, and the missing data is different each time.  What could cause this behavior? 

Environment

Release : 15.x

Component : CA PPM DATA WAREHOUSE

Resolution

Missing data in Data Warehouse can happen if the data was updated in the meantime whilst the job was running. This is due to the fact that last_updated_date is used to move all updated fields to DWH. When the job starts it takes the job start date, and compares to last_updated_date on the data it imports. Anything that's updated after the job start date will not get imported.

So the scenario you are mentioned is possible if the Full was running very long time and someone modified a project = this project will not be in DWH.

We have a feature created with Product Management to improve this behavior. To vote for it, please bring it up on the Innovation calls, this will help prioritize it.