TDM: MongoExecutionTimeoutException while running PII Scan against MongoDB database

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

When are running the PII scan against a MongoDB database we are getting below error on some of the databases while the others are working fine.

"Failed to gather metadata”

 We are on TDM Portal 4.11.71

 

      c.c.t.c.j.JobEngine:  sendJobStatus: sending status to job engine: jobId: 16463 status: Failed info: PII scan of table BookingRunConfig failed: Exception raised: MongoExecutionTimeoutException: Executor error during getMore :: caused by :: operation exceeded time limit
2024-07-11 18:12:42.529 UTC [INFO ] [ModellingEngine-MODELPIISCAN-jobId-16463] --- [U:][M:][P:]                      c.c.t.a.s.TDMAuditLogService:  insertLog: projectId: 2426 versionId: 2427 user: subba status: FAILED desc: PII scan of table BookingRunConfig failed: Exception raised: MongoExecutionTimeoutException: Executor error during getMore :: caused by :: operation exceeded time limit
2024-07-11 18:12:42.547 UTC [INFO ] [ModellingEngine-MODELPIISCAN-jobId-16463] --- [U:][M:][P:]                           c.c.t.e.ModellingEngine:  shutdownThreadpool: job: 16463:pool: modelling engine worker thread pool: cleaning up now
2024-07-11 18:12:42.547 UTC [INFO ] [pool-34-thread-1         ] --- [U:][M:][P:]                      c.c.t.e.ModellingJobProgress:  run: modelling engine progress thread:  shutting down: sending final notification for job: 16463 state: FAILED
2024-07-11 18:12:42.547 UTC [INFO ] [ModellingEngine-MODELPIISCAN-jobId-16463] --- [U:][M:][P:]                           c.c.t.e.ModellingEngine:  run: job: 16463: scan errors: [PII scan of table BookingRunConfig failed: Exception raised: MongoExecutionTimeoutException: Executor error during getMore :: caused by :: operation exceeded time limit]
2024-07-11 18:12:42.548 UTC [INFO ] [ModellingEngine-MODELPIISCAN-jobId-16463] --- [U:][M:][P:]                           c.c.t.e.ModellingEngine:  run: job: 16463: ModellingEngine run complete: final state: FAILED run time: 4.6 mins

Environment

 TDM Portal 4.11.71 and up

Cause

MongoExecutionTimeoutException => this is been thrown by the mongo driver

Resolution

We have a patch  /TDMWeb-4.11.99.0.zip  that has a parameter to increase the Timeout for reading into MongoDB.

You must add/change the following parameter into application.properties:

tdmweb.profiling.query.timeout=

This parameter must be defined in seconds. Ex: 60 to 1 minute, 120 to 2 minutes and so on.

The default value is 60.