TDM: Masking jobs running super slow - to start masking
Article ID: 190210
Updated On:
Products
Issue/Introduction
On a smaller scale, our masking jobs were running better after upgrading to 4.9. However, masking on large tables is very slow. Every time we 'restarted' a job after cancellation or failure, it has been super slow to start. PFA a screen shot ... the job has been running for over 1 hour 50 minutes and still it has not began to mask any rows. We are using TDM Portal 4.9.0.12 and FastDataMasker-4.9.0.0. Our source and target databases are running on the same Oracle schema.
To give you an idea, it took over 20 hours to mask 111,963,468 rows.
Environment
Release : 4.9
Component : CA Test Data Manager
Resolution
We recommend you upgrade to the latest Portal and FDM patched releases, which at the time this KB was written is FastDataMasker-4.9.18.0 and TDMWeb-4.9.37.0.
After upgrading, you should use block masking to help increase performance on large jobs.
Suppose you have a table with 100,000,000 rows and you want to run masking jobs in parallel to mask all the rows.
When activated through ... LARGETABLESPLITENABLED=Y
... FDM will create 10 threads(default setting) to process all the blocks that were created by dividing the 100 million rows into chunks. Each block of the size LARGETABLESPLITSIZE=5000000 will be processed.
When a thread completes processing a block, it will process the next block from the queue.
This thread pool is configurable by setting the option PARALLEL=n
We have tried to run it so far with the settings with larger batch size 30k, Fetch Size 30K and optimally up to 4.5 million as LargeTableSplitSize. However, even with smaller split size, maximum four threads are possible, as back-end goes out of space for more number of thread.
As a guideline to the kind of space that must be maintained in the back-end in the Default Table Space and Temp Space, work with your DBA to analyze your table space used and multiple it by 2 times the number of threads you plan to to use.
Additional Information
You can find the Masking Options Reference at https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/continuous-testing/test-data-management/4-9/reference/masking-functions-and-parameters/masking-options.html
FastDataMasker-4.9.22.0.zip resolves the issue. Please contact Support for this version or a later one.
Feedback
