PII scan performance issue on TDM docker portal

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

PII scan is very slow and the job failed after approximately 10 hours of execution.

There are about 264 tables, and the PII scan level is at 50%.

The cpu usage of tdmweb pod is 100%.

The log shows a lot of "Failed to obtain JDBC Connection" error:

c.c.t.j.c.JobEngineServiceController:  Updating status for requestId xxx : JobStatusRequest [status=Failed, statusMessage=PII scan of table xxx failed: Exception raised: : Failed to obtain JDBC Connection: PII profiler failed with an exception: table: xxx, artifactLocation=null]

Environment

TDM docker portal 4.11

Resolution

1. Increase cpu of the tdmweb pod to 16
2. Increase JDBC connection pool to 100
   Add application property spring.datasource.tomcat.max-active=100 in values.yaml file, for example:
   Before:
   

   applicationProperties: "tdmweb.TDMMaskingService.taskTimeout=30|tdmweb.profiling.uncommitted.reads=true|tdmweb.profiling.query.timeout=300"

   After:
   

   applicationProperties: "tdmweb.TDMMaskingService.taskTimeout=30|tdmweb.profiling.uncommitted.reads=true|tdmweb.profiling.query.timeout=300|spring.datasource.tomcat.max-active=100"

3. Restart tdmweb pod

 

After these settings, the PII scan runs very smoothly, and the job is completed successfully.