Mask throttling mechanism of TDM docker portal

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

How does the mask throttling work, and how it interacts with KEDA?

Time-based Throttling:

Is there any way in our existing helm-based deployment to throttle masking jobs depending on the time of day?

We may have a requirement from our network team to set different limits on network throughput for both during the day and during at night.

 

Environment

TDM docker portal on openshift

Resolution

1. The Four Throttling Mechanisms
TDM employs a layered approach to throttling to ensure both the application and your target databases remain stable under heavy loads.

* Pod-Level Throttling (`poolSize`): This limits how many concurrent masking tasks a single masking pod can execute at once.
In your `values.yaml`, this is controlled by the `.Values.masking.service.poolSize` value, which is injected into both the masking and tdmweb pods.
Because the masking containers do the heavy lifting, this number must be balanced against the CPU and memory limits you assign to those specific pods.

* Job-Level Throttling: This is governed by the `.Values.tdmweb.maximumNumberOfConcurrentMaskingTasksPerMaskingJob` Helm value, which directly controls the `TDMWEB_TDM-MASKING-SERVICE_MAXIMUM_NUMBER_OF_CONCURRENT_MASKING_TASKS_PER_MASKING_JOB` environment variable you referenced.
It is injected into the `tdmweb` container, which acts as the orchestrator.
It applies globally to a specific job across the entire cluster to prevent a single massive job from overloading the backend database and causing starvation of other smaller masking jobs waiting in the queue.
TDM uses ZooKeeper to enforce this limit and track active tasks across all worker pods.

* System-Wide Throttling: This is the system-wide property `tdmweb.tdm-masking-service.maximum_number_of_concurrent_masking_tasks` (which defaults to 32).
It acts as a global ceiling for the entire TDM Portal, regardless of how many separate jobs are running.
You can easily configure this in your current Helm deployment without needing a new version from Broadcom.
You just need to append it to the `applicationProperties` string within both the `tdmweb` and `masking` sections of your `values.yaml`.

* Connection Profile Throttling: You can also govern throughput at the database connection level.
When creating or editing a Connection Profile in the TDM Portal UI, you can expand the "Extra Options" section to find a specific setting for "Maximum number of concurrent masking tasks".
This is incredibly useful if you have a specific legacy database that requires stricter query limits than your standard infrastructure.

For a full breakdown of these features, you and your team can reference the official documentation here:
Manage Throttling
https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-11/provisioning-test-data/mask-data-with-ca-tdm-portal/manage-throttling.html


2. Interaction with KEDA Autoscaling

There is no conflict between your TDM throttling variables and your KEDA limits.
They actually work together seamlessly to protect your infrastructure.

KEDA is event-driven. In our architecture, the custom metric returned to the KEDA external scaler is `numberOfActiveMaskingTasks`.
The `tdmweb` orchestrator uses your throttling variables to determine exactly how many tasks are allowed to be active at any given moment.
KEDA simply polls `tdmweb` for that active task count and scales the worker pods up or down to accommodate the approved workload.

Furthermore, the `autoscaling.scaledObject.maxReplicaCount` Helm value is passed directly to the Horizontal Pod Autoscaler (HPA) definition that KEDA creates.
This holds the absolute maximum number of replicas for your masking deployment.
This serves as another layer of protection against over-utilization of system resources, indirectly capping the maximum number of concurrent masking tasks your cluster can physically process.


3. Time-Based Throttling
Currently, TDM does not natively support time-based throttling at the application level.
However, you have a perfect workaround available right now.
You can manage this completely outside of the Broadcom TDM Helm chart by deploying a standard Kubernetes `CronJob` to act as your time-based trigger.

You can apply a separate Kubernetes `CronJob` to your OpenShift cluster that automatically patches the existing KEDA `ScaledObject` to lower the `maxReplicaCount` during business hours.
By artificially limiting the maximum number of worker pods during the day, you inherently cap the network throughput and database load to satisfy your network team's requirements.

Here is a YAML snippet showing how to easily append this Cron trigger outside of the Helm chart to scale down to a maximum of 2 pods every morning at 8:00 AM:
===============================
apiVersion: batch/v1
kind: CronJob
metadata:
  name: tdm-daytime-throttle-trigger
  namespace: <your-namespace>
spec:
  schedule: "0 8 * * 1-5" # Runs at 8:00 AM, Monday through Friday
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: tdm
          restartPolicy: OnFailure
          containers:
          - name: keda-patcher
            image: bitnami/kubectl:latest
            command:
              - /bin/sh
              - -c
              - "kubectl patch scaledobject masking --type='merge' -p='{\"spec\":{\"maxReplicaCount\": 2}}'"
===============================

You would simply deploy a second identical `CronJob` scheduled for your evening window (e.g., 6:00 PM) that patches the `maxReplicaCount` back up to your normal high limit (like 100) for overnight batch processing.
This keeps your TDM deployment completely vanilla while fulfilling the dynamic network requirements.