Resolve Data Aggregator REST API 503 errors caused by concurrent request limits

Products

• Network Observability
• CA Performance Management

Issue/Introduction

The Data Aggregator (DA) REST/OpenAPI service returns HTTP 503 errors under concurrent request load. The application log records entries similar to:

Request rejected. Could not acquire a permit within 30000ms. The server is at its concurrent request limit.

This occurs when the number of simultaneous requests to the Data Aggregator exceeds the configured concurrency limit.

Environment

• Product: DX NetOps Performance Management — Data Aggregator
• Version: 25.4.x (Spring Boot-based Data Aggregator)
• OS: Linux

Cause

The Data Aggregator limits the number of concurrent REST/OpenAPI requests it will process at once, using a request-limiter setting (maxRequests). When the number of concurrent callers exceeds this limit, additional requests wait up to the configured timeout and are then rejected with a 503 error. If legitimate concurrent traffic (for example, polling from an integrated system) regularly exceeds the configured limit, rejections will recur until the limit is raised to match actual usage.

Resolution

1. Back up the current configuration file before editing:

   cp /app/CA/IMDataAggregator/config/com.ca.im.odata.filters.OpenAPIRequestLimiterFilter.cfg \
      /app/CA/IMDataAggregator/config/com.ca.im.odata.filters.OpenAPIRequestLimiterFilter.cfg.bak
   

2. Open com.ca.im.odata.filters.OpenAPIRequestLimiterFilter.cfg in a text editor. It contains:

   maxRequests = 4
   suspendMs = -1
   waitMs = 30000
   

3. Increase maxRequests to a value that comfortably covers expected concurrent usage. Increase gradually — for example, from 4 to 12 — rather than setting a very high value, since raising this limit increases concurrent load on the Data Aggregator's CPU, memory, and database connections.

   maxRequests = 100
   suspendMs = -1
   waitMs = 30000
   

4. Save the file. The Data Aggregator watches this configuration file and reloads it automatically. Confirm the change took effect by checking the application log for:

   Request limiter semaphore (re)initialized with  permits.
   

5. If the new value is not picked up automatically, restart the Data Aggregator service:

   systemctl restart dadaemon
   

6. Confirm the fix by monitoring the application log for the "Request rejected" message during normal peak usage. If rejections continue, increase maxRequests further, in moderate increments, and re-test.