A Test Data Manager (TDM) masking job may show a status of "Completed" in the TDM Portal shows as failed

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

A Test Data Manager (TDM) masking job processing a large number of rows (e.g., 11+ million) fails abruptly or terminates before all rows are processed. The job may show a status of "Completed" in the TDM Portal even though the audit files indicate that only a portion of the expected rows were successfully masked.

Environment

• TDM Portal 4.11.x 
• Fast Data Masker (FDM)

Cause

1. Engine Termination: The FDM Engine terminates abnormally due to system resource constraints, such as an Out of Memory condition (Exit Code 137). In some versions, the Portal incorrectly reports this as a successful completion. 
2. Resource Bottlenecks: High-volume tables can exhaust database temp space or default table space when processed without optimization. 
3. Rule Configuration: Masking rules containing WHERE clauses that are improperly sequenced can cause subsequent rules to be ignored or the job to behave unexpectedly.

Resolution

1. Address Memory and Engine Stability

• Verify FDM logs for Exit Code 137.
• If found, upgrade to TDMWeb-4.10.199.0 or later to ensure the Portal correctly handles and reports engine terminations.
• Increase the memory allocation for the FDM executable.

2. Optimize for Large Tables

To improve stability and performance on tables with millions of records:

• Enable large table splitting by setting LARGETABLESPLITENABLED=Y in the masking options.
• Define a LARGETABLESPLITSIZE (e.g., 5,000,000) to process the data in manageable chunks.
• Utilize PARALLEL=n to enable multi-threaded processing.

   IMPORTANT: Work with your DBA to ensure the database Temp Space and Table Space are at least 2x the number of parallel threads being used.

3. Correct Masking Rule Sequence

• Review the masking CSV file.
• Ensure that all rules containing a WHERE clause are positioned at the bottom of the file, following all rules that do not use a WHERE clause. Tables not getting masking despite JOB completion

4. Refresh Seed Data

• If the job fails with NULL insertion errors, reload the seed list data into the Scramble database to ensure data integrity.