TDM: Masking jobs running super slow - to start masking
Article ID: 190210
Updated On:
Products
Issue/Introduction
On a smaller scale, masking jobs were running better after upgrading to 4.9. However, masking on large tables is very slow. Every time a job was 'restarted' after cancellation or failure, it has been super slow to start. PFA a screen shot ... the job has been running for over 1 hour 50 minutes and still it has not began to mask any rows.
Environment
TDM Portal 4.9.0.12 and
FastDataMasker-4.9.0.0.
Source and target databases are running on the same Oracle schema.
Resolution
Recommend to upgrade to the latest 4.9 Portal and FDM patched releases. The Patches can be accessed from this link: Test Data Manager (TDM) Support Patches
After upgrading, use block masking to help increase performance on large jobs.
With having a table of 100,000,000 rows and wanting to run masking jobs in parallel to mask all the rows, when activated through LARGETABLESPLITENABLED=Y, FDM will create 10 threads (default setting) to process all the blocks that were created by dividing the 100 million rows into chunks. Each block of the size LARGETABLESPLITSIZE=5000000 will be processed.
When a thread completes processing a block, it will process the next block from the queue.
This thread pool is configurable by setting the option PARALLEL=n
Ran it so far with the settings with larger batch size 30k, Fetch Size 30K and optimally up to 4.5 million as LargeTableSplitSize. However, even with smaller split size, maximum four threads are possible, as back-end goes out of space for more number of thread.
As a guideline to the kind of space that must be maintained in the back-end in the Default Table Space and Temp Space, work with a DBA to analyze the table space used and multiple it by 2 times the number of threads planning to to use.
Additional Information
You can find the Masking Options Reference at https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/continuous-testing/test-data-management/4-9/reference/masking-functions-and-parameters/masking-options.html
FastDataMasker-4.9.22.0.zip resolves the issue. Please contact Support for this version or a later one.
Feedback
