Optimizing and Troubleshooting Masking Job Failures in CA Test Data Manager

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

You encounter a failure when trying to mask the whole database.

Environment

TDM docker portal 4.11

Cause

The masking job failure typically occurs due to one or more of the following factors:

• Unverified PII items or incorrect masking functions assigned to columns.
• Suboptimal performance configurations for large-scale masking workloads.
• Insufficient masking service pool size -- extra task(s) need to wait for running tasks to complete, which might lead to timeout issue.
• Parallel masking will fail on a table without primary key
• Masking job takes too long time, although it completes successfully, the job is still masked as failed due to JWT expire (it's one hour by default)
• For big table, query takes too long time causing query timeout.

Resolution

You can resolve these failures by following these optimization and verification steps:

1. Verify PII and Masking Functions

Before you initiate the masking job, ensure you review and confirm all PII items and assigned masking functions. Incorrect function assignments (such as applying FORMATENCRYPT to a DATETIME column) cause validation and execution errors.

2. Optimize Masking Performance

You should optimize your masking options to ensure efficient data processing. Refer to the standard performance optimization guidelines in the CA TDM Portal to configure split sizes and concurrent table masking appropriately.

Masking Performance Optimization in CA TDM Portal

3. Increase Masking Service Pool Size

If your job contains a high number of tasks, you must ensure the masking pod service pool is large enough to handle them. Increase the masking.service.poolSize in values.yaml to a value greater than the maximum number of tasks defined in your masking job to prevent extra masking tasks pending on running tasks.

4. Run in Preview Mode

You should run the masking job in preview mode before actual execution. This allows you to verify the masking configuration and identify potential data type mismatches or primary key requirements for parallel processing without modifying target data.

 

5. Increase jwt.refreshInterval

jwt.refreshInterval=3600 by default, increase it if the masking job takes longer time than that. 

This is application property of TDM, add it to tdmweb.applicationProperties in values.yaml file.

e.g. set jwt.refreshInterval=21600

applicationProperties: "tdmweb.TDMMaskingService.taskTimeout=30|tdmweb.profiling.uncommitted.reads=true|tdmweb.profiling.query.timeout=300|jwt.refreshInterval=21600"

 

6. Increase jdbc query timeout

It depends on the jdbc driver.

For example, for sqlserver jdbc, we can add additional parameters in connection profile such as:

queryTimeout=0;socketTimeout=0;cancelQueryTimeout=300 

This will set the query timeout to be unlimited.

 

Additional Information

Masking Performance Optimization in CA TDM Portal