Understanding TOTAL_ROWS_TO_MASK in TDM Masking Audit Reports

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

When reviewing a masking audit file in CA Test Data Manager (TDM), the TOTAL_ROWS_TO_MASK value appears higher than the actual number of rows in the source database table. Why is this occurring?

Resolution

The TOTAL_ROWS_TO_MASK value is not a direct representation of the table's total row count. Instead, it represents the calculated number of records that the masking job is targeted to process based on your specific masking job configuration.

Several factors contribute to this value:

• Filters: If your masking configuration includes filters (e.g., WHERE conditions), the TOTAL_ROWS_TO_MASK reflects the number of records meeting those criteria.
• Configuration Complexity: When multiple masking functions are applied to different columns, or multiple functions are applied to the same column, the masking job generates multiple queries. The TOTAL_ROWS_TO_MASK value is derived from the sum of the result sets of these queries. Consequently, this sum could be higher than the actual row count of the table.
• Purpose: The primary purpose of TOTAL_ROWS_TO_MASK is to provide a metric for tracking masking progress by comparing processed rows against total rows to mask.

Verification Steps: If you need to verify the audit report data, follow these steps to isolate the specific queries:

1. Enable Diagnostics: Set the DIAGLEVEL parameter to 2 or above for your masking job.
2. Run the Job: Execute the preview or the actual masking job.
3. Inspect Logs: Review the corresponding FDM log to identify the queries executed during the process.
   Sample of fdm log:
   

   2026-08-25 18:31:35.217 [pool-4-thread-1] INFO - SQL select statement =SELECT "FirstName","ID" FROM "dbo"."Person" WHERE id > 10000 order by "ID"
   2026-08-25 18:31:35.219 [pool-4-thread-1] INFO - canCommit = true
   2026-08-25 18:31:35.219 [pool-4-thread-1] INFO - Fetchsize = 500000
   2026-08-25 18:31:35.219 [pool-4-thread-1] INFO - Autocommit = false
   2026-08-25 18:31:35.224 [pool-4-thread-1] INFO - Person executed select query
   2026-08-25 18:31:35.224 [pool-4-thread-1] INFO - preparedUpdateStatement = update "dbo"."Person" SET "FirstName" = ? WHERE "ID" = ?
   2026-08-25 18:31:51.888 [pool-4-thread-1] INFO - Person processed 500000 rows
   2026-08-25 18:32:08.572 [pool-4-thread-1] INFO - Person processed 1000000 rows
   ...

4. Validate: Rewrite these queries using SELECT COUNT(*) and run them directly against your database.
   For example: change above select statement to: 
   

   SELECT count(*) FROM "dbo"."Person" WHERE id > 10000 

5. Compare: Sum the results of these queries. This sum should match the TOTAL_ROWS_TO_MASK value recorded in your audit report.