Resolve dadaemon REST API 503 errors caused by rapid DELETE requests in DX NetOps Performance Management

Products

• Network Observability
• CA Performance Management

Issue/Introduction

When sending high-frequency REST DELETE requests to the dadaemon API, the service may return an HTTP 503 Service Unavailable error. This behavior occurs because back-to-back API calls can overwhelm the system, causing temporary instability. This article provides steps to mitigate these errors by pacing requests and adjusting timeout settings.

Symptoms

• REST API DELETE calls return HTTP 503 errors.
• System stability improves after restarting dadaemon but reverts to errors following bursts of DELETE activity.
• Intermittent system unavailability observed under high API request load.

Environment

• DX NetOps Performance Management (all supported releases)
• Data Aggregator environment experiencing REST API performance bottlenecks.

Cause

High-frequency, back-to-back REST DELETE requests can cause the Data Aggregator to exceed processing thresholds, leading to temporary service unavailability (503). Additionally, default Kafka metadata wait times may be too high, allowing requests to block system resources.

Resolution

To resolve this issue, implement request pacing and optimize Kafka timeout settings:

1. Implement Request Pacing: Ensure that applications issuing REST DELETE calls implement a delay between requests. Avoid sending back-to-back requests. Pacing helps the dadaemon process manage the cleanup load without exhausting system resources.

2. Adjust Kafka Timeout Settings: Edit the Kafka producer configuration to reduce the time requests spend waiting for metadata, which allows requests to fail-fast rather than blocking the service:

   1. Navigate to the configuration directory: /apps/IMDataAggregator/config/
   2. Open kafka.InventoryProducer.cfg in a text editor.
   3. Add or update the following properties:
      • max.block.ms=3000
      • request.timeout.ms=5000
   4. Save the changes.

3. Restart dadaemon: Restart the dadaemon service to apply the configuration updates: /etc/init.d/dadaemon restart

4. Verify: Monitor the REST API for 503 errors. The system should remain stable under the managed request flow.

Additional Information

• If the issue persists, review the dadaemon logs for additional error details.
• Ensure that no other unauthorized changes have been made to the com.ca.im.odata.filters.OpenAPIRequestLimiterFilter.cfg file, as incorrect configuration here can also impact API performance.