search cancel

PIPE Delimited File Obfuscation Performance for FDM

book

Article ID: 215736

calendar_today

Updated On:

Products

CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

We are using FDM 4.9.155.0 to mask a very large PIPE delimited file. The file is roughly 14GB in size. It contains 3,000,000 records; around 1,000 columns of which we are masking only 10. We are using WHERE clauses to split the table, and have noticed the masking takes days to complete. The rate we are seeing is 10,000 records /1 minute. We would like to know how to increase performance, and speed up the masking, as this is impacting our delivery.

Environment

Release : 4.9.1

Component : CA Test Data Manager - Fast Data Masker

Windows Server 2012R2

8 Processor CPU and 32 GB RAM

Cause

FDM does not support Large Table Split Size or Parallel processing for masking flat files. The only option would be to play around with the CHUNKSIZE option, which controls the number of lines that are written at a time to the file masking. 

Resolution

 
 

Upping the CHUNKSIZE from 10,000 to 100,000 might help with the time taking to write the file.

An enhancement request has been submitted to add Large Table Split processing to help improve the performance of large flat-file obfuscation.

The FDM enhancement can be tracked through US751815