FDM Issue with Large Tables

book

Article ID: 130572

calendar_today

Updated On:

Products

CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

When we try to mask a huge table on Oracle containing 50 million records which only uses the format encrypt function, the masking is getting stalled after the first set of fetch records.  If i specify 1000 as the fetch size, then it will not do anything after the first 1000 records. If we create the exact same table with lesser number of records, such as 100K, the masking works correctly. Looks like FDM is not masking for tables with 8 million rows as well.  We are using FDM version 4.7.3207.0.

Cause

When looking through this initial description, it did look like it had to do with a large row count.  BUT another way to look at this is that in the larger row count, there was bad data.  In this case, the DATA was not bad, just NULL.  A previous fix had to be backed out and a new variable is introduced with the fix release to ensure both reported cases are resolved.
There was only a unique index on the table but no primary key. When we re created the table with a primary key it started working so not sure if that's the exact problem.
I am looking to perform some more tests.
**********************************
If memory is an issue when masking large object then set the following options to a small value
FETCHSIZE=100
BATCHSIZE=100
COMMIT=100
**********************************

The above sizes did not fix the issue.  Memory is not the issue.  Has to be the NULL in the data.  Need a Code Change.

Environment

FDM version 4.7.3207.0

Resolution

A work around presented to a user that worked (names obfuscated):
Make ORIGINAL_KEY column as non nullable
or
drop UNIQUE INDEX "TDM_NAME" on COMPANY_TEMP table

The above work around(s) worked.  This verified that Development needed to change the code.

FastDataMasker-4.7.3220.0.zip has been created to resolve this issue.  Please Note that while fixing this defect, we had to redo DE383181 (01179590-Unknown error message in FDM) To make DE383181 work, the customer will have to set option:
 
UPDATENULLINDEXCOLUMN=Y

Default for the vast majority of the standard cases is

UPDATENULLINDEXCOLUMN=N

Please contact Broadcom support for the FastDataMasker-4.7.3220.0.zip file (or later).  If installing a GA release after 05APR2019, please contact Broadcom Support to verify if this fixed version made it into your GA Release version.