Error: dslPluginFailure timeout during bulk API operations in Continuous Delivery Director SaaS

Products

• Continuous Delivery Director SAAS

Issue/Introduction

Intermittent outages or timeouts occur when performing bulk operations via the Continuous Delivery Director (CDD) REST API.
Users see errors related to dslPluginFailure and timeouts in the response object while the system experiences high latency.

Symptoms

• API responses return HTTP 422 or 500 errors.
• The error message contains: {"type":"general","messageCode":"dslPluginFailure","arguments":[{"name":"pluginErrorMessage","value":"FAILED-Failed to contact proxy - got timeout..."}]}.
• Application logs show database lock wait timeouts: SqlExceptionHelper: Lock wait timeout exceeded; try restarting transaction.
• High frequency of DELETE or POST requests to the /dsl-manifests or /releases endpoints from a single source.

Cause

A high volume of concurrent API requests, such as a "bulk DELETE storm," saturates database transactions.
When multiple requests attempt to modify releases with active phases, the database holds locks for extended periods, leading to timeouts for subsequent requests.

Resolution

To resolve this issue, optimize how the external system interacts with the CDD API:

1. Reduce the concurrency of API requests to avoid saturating database transactions.
2. Implement a retry mechanism with exponential backoff for HTTP 422 errors.
3. Ensure releases do not have active phases before attempting a bulk DELETE operation.
4. Space out large batches of API calls (e.g., allow 1-2 seconds between requests) to allow the database to clear pending transactions.