PIPE Delimited File Obfuscation Performance for FDM

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

We are using FDM 4.9.155.0 to mask a very large PIPE delimited file. The file is roughly 14GB in size. It contains 3,000,000 records; around 1,000 columns of which we are masking only 10. We are using WHERE clauses to split the table, and have noticed the masking takes days to complete. The rate we are seeing is 10,000 records /1 minute. We would like to know how to increase performance, and speed up the masking, as this is impacting our delivery.

Environment

Release : 4.9.1

Component : CA Test Data Manager - Fast Data Masker

Windows Server 2012R2

8 Processor CPU and 32 GB RAM

Cause

FDM does not support Large Table Split Size or Parallel processing for masking flat files. The only option would be to play around with the CHUNKSIZE option, which controls the number of lines that are written at a time to the file masking. 

Resolution

 

 

Upping the CHUNKSIZE from 10,000 to 100,000 might help with the time taking to write the file.

An enhancement request has been submitted to add Large Table Split processing to help improve the performance of large flat-file obfuscation.

The FDM enhancement can be tracked through US751815