PIPE Delimited File Obfuscation Performance for FDM
search cancel

PIPE Delimited File Obfuscation Performance for FDM


Article ID: 215736


Updated On:


CA Test Data Manager (Data Finder / Grid Tools)


We are using FDM to mask a very large PIPE delimited file. The file is roughly 14GB in size. It contains 3,000,000 records; around 1,000 columns of which we are masking only 10. We are using WHERE clauses to split the table, and have noticed the masking takes days to complete. The rate we are seeing is 10,000 records /1 minute. We would like to know how to increase performance, and speed up the masking, as this is impacting our delivery.


Release : 4.9.1

Component : CA Test Data Manager - Fast Data Masker

Windows Server 2012R2

8 Processor CPU and 32 GB RAM


FDM does not support Large Table Split Size or Parallel processing for masking flat files. The only option would be to play around with the CHUNKSIZE option, which controls the number of lines that are written at a time to the file masking. 



Upping the CHUNKSIZE from 10,000 to 100,000 might help with the time taking to write the file.

An enhancement request has been submitted to add Large Table Split processing to help improve the performance of large flat-file obfuscation.

The FDM enhancement can be tracked through US751815