Concurrent Masking Job Failure

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

When you run concurrent masking jobs in Test Data Manager (TDM), multiple masking jobs may fail at the same time with exceptions. This often occurs when resources are insufficient to handle the concurrent load.

Symptoms

• Concurrent masking jobs fail simultaneously with exceptions.
• You observe high CPU and memory utilization (reaching 100%) on masking pod.
• FDM log shows the masking job(s) recieved interrupt request.

[Thread-1] INFO - Interrupt request detected - the masking will be stopped

Cause

The masking service pool size is set to 10 (4 by default) in previous performance tests while the cpu/mem are still using the default values.

The masking container may excceed the resource boundaries during concurrent masking tasks, then openshift kills and restarts the container.

(OpenShift will instantly kill and restart a container for exceeding memory limits, but it should only throttle performance for exceeding CPU limits.)

Resolution

Change the masking pool size back to the default value 4, the concurrent masking jobs are completed successfully.

It's recommended to keep the default values as masking pod can be auto scaled up.