TDM: Masking jobs running super slow - to start masking
search cancel

TDM: Masking jobs running super slow - to start masking

book

Article ID: 190210

calendar_today

Updated On:

Products

CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

On a smaller scale, masking jobs were running better after upgrading to 4.9. However, masking on large tables is very slow. Every time a job was 'restarted' after cancellation or failure, it has been super slow to start. PFA a screen shot ... the job has been running for over 1 hour 50 minutes and still it has not began to mask any rows. 


Environment

TDM Portal 4.9.0.12 and

FastDataMasker-4.9.0.0.

Source and target databases are running on the same Oracle schema.

Resolution

Recommend to upgrade to the latest 4.9 Portal and FDM patched releases.  The Patches can be accessed from this link: Test Data Manager (TDM) Support Patches

After upgrading, use block masking to help increase performance on large jobs. 

With having a table of 100,000,000 rows and wanting to run masking jobs in parallel to mask all the rows, when activated through LARGETABLESPLITENABLED=Y,  FDM will create 10 threads (default setting) to process all the blocks that were created by dividing the 100 million rows into chunks. Each block of the size LARGETABLESPLITSIZE=5000000 will be processed.

When a thread completes processing a block, it will process the next block from the queue.

This thread pool is configurable by setting the option PARALLEL=n

Ran it so far with the settings with larger batch size 30k, Fetch Size 30K and optimally up to 4.5 million as LargeTableSplitSize. However, even with smaller split size, maximum four threads are possible, as back-end goes out of space for more number of thread.

As a guideline to the kind of space that must be maintained in the back-end in the Default Table Space and Temp Space, work with a DBA to analyze the table space used and multiple it by 2 times the number of threads planning to to use.

 

Additional Information

For more information on splitting the tables and parallel threads, see the TDM 4.9 'Fast Data Masker Best Practices' at https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/continuous-testing/test-data-management/4-9/provisioning-test-data/mask-production-data-with-fast-data-masker/fast-data-masker-best-practices.html#concept.dita_ce0565291bdd20a964468136ca6730a3ba4e2531_splittables

You can find the Masking Options Reference at https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/continuous-testing/test-data-management/4-9/reference/masking-functions-and-parameters/masking-options.html

FastDataMasker-4.9.22.0.zip resolves the issue.  Please contact Support for this version or a later one.