search cancel

Encountering an "API Plan Limit Exceeded" Error Even When Traffic Is Well Underneath the Set Rate Limit.

book

Article ID: 8735

calendar_today

Updated On:

Products

CA API Developer Portal CA API Gateway

Issue/Introduction

When load testing (or very high traffic to) an API call which uses the Rate Limit assertion set by an API Plan or Account Plan in the CA API Developer Portal ("Portal"), the following error may be encountered even when traffic is well underneath the value set for the Rate Limit assertion:

API Plan Limit Exceeded

Environment

  • Portal version 3.5 and lower, and CA API Gateway ("Gateway") version 9.2 and lower.

Cause

  • When a Rate Limit is configured on an API Plan or Account Plan in the Portal, it will cause API calls to be throttled to stay underneath the value set for the Rate Limit
  • By default, the Portal will configure the Rate Limit assertion (found in the associated policy viewable in Policy Manager) with the value chosen earlier for the Maximum Requests Per Second function, and have the Spread Limit Over X Sec Window function disabled, and an action set to Throttle. This does not normally cause issues with typical traffic volumes, but depending on the frequency of traffic/API calls (especially if it's being load tested), some requests can be throttled in order to stay under the value set for the Maximum Requests Per Second function, negatively impacting the API client sending the request.
  • The Rate Limit Assertion does some math for the Maximum Requests Per Second, which is not always well understood. If the value is set to 100 (for example), then the Rate Limit assertion will process all traffic as expected as long as there is no more than 1 request per 1/100th of a second. If two calls come in at under 1/100th of a second apart (which often happens when using load testing tools) for example, then the second request will be throttled. If a request is throttled, it is considered a failure case and will throw the error to the client.

Resolution

If there is a high traffic load (i.e. a load test running), and there is an expectation that it still be handled as it is viewed as under the Maximum Requests Per Second value, then it is recommended to enable the Spread Limit Over X Sec Window function on the Rate Limit assertion. The Spread Limit Over X Sec Window function allows for bursts of traffic, which includes traffic coming in under the buffer that is set by the value for Maximum Requests Per Second which is often the case with load tests. The Gateway product documentation (more details found in the Additional Information section below) explains the following for the Spread Limit Over X Sec Window function on the Rate Limit assertion:

  • What it does: "Determines whether to allow a burst of requests to be spread across a window of time or whether to enforce a hard cap."
  • When the Spread Limit Over X Sec Window is disabled, then "the Gateway only accepts requests arriving no sooner than 1/limit of a second. For example, if the Max requests per second is 100, at least 1/100 second must have elapsed between requests. Requests that arrive sooner are either throttled or shaped (based on the "When limit exceeded" setting). Disallowing burst traffic is recommended only for advanced users."
  • It also has a note which reads: "It is not recommended to disable burst traffic on a counter that will be servicing multiple concurrent requests, particularly at high rates. Doing so can lead to unintended throttling or delaying of multiple requests that arrive at exactly the same time."

The recommended resolution is to enable the Spread Limit Over X Sec Window function on the Rate Limit assertion, typically with a value of 60 seconds or higher. The gotcha is that the Portal does not automatically enable that function, however it can be configured to after a couple of file changes and a restart of the Portal service. This can be achieved by following the steps below on the Portal:

  1. Connect to the Portal via SSH to get a terminal, or connect on a console level.
  2. Edit the following file using vi: /opt/Deployments/lrs/server/webapps/ROOT/templates/apiplan.xml
    1. Change <L7p:HardLimit booleanValue="true"/> to <L7p:HardLimit booleanValue="false"/>
  3. Edit the following file using vi: /opt/Deployments/lrs/server/webapps/ROOT/templates/accountplan.xml
    1. Change <l7:HardLimit>true</l7:HardLimit> to <l7:HardLimit>false</l7:HardLimit>
  4. Restart the Portal for the changes to take effect.
  5. Sync the API Plans and Account Plans using the buttons on the Layer 7 Gateway plugin page in the /admin section of the Portal website to initiate a sync.
  6. Refresh the Policy Manager to the Gateway and view the changes. The Rate Limit assertion should now have the Spread Limit Over X Sec Window function enabled.

After the changes above, all new API Plans or Account Plans will also have that automatically applied. This is a useful resolution if the Spread Limit Over X Sec Window function is required in the environment, particularly if developers do not have access to Policy Manager to make the changes manually.


*** The above is for portal 3.5, for portal 4.x there is a schedule task "portal sync account plan".


https://api-broadcom-ca.wolkenservicedesk.com/es/attachments/get_attachment_content?uniqueFileId=500787650366


On "portal sync account plan" policy window, search by "hardlimit" to find the "set context variable rateLimitXml" assertion.

Open this assertion and change the 

<l7:HardLimit>true</l7:HardLimit>

to

<l7:HardLimit>false</l7:HardLimit>


"Save and Activate" the policy, make some changes on current account plan, after next sync, open the "Account Plans Fragment" policy, search with "rate limit", the "Apply rate limit" assertion should now have option "Spread limit over 60 sec windows" checked, 

https://api-broadcom-ca.wolkenservicedesk.com/es/attachments/get_attachment_content?uniqueFileId=500787808037


As soon as this option is checked, it should resolve the problem.

Additional Information

Attachments

Powered by Wolken Software