NSX API throws error message: "Client '######' exceeded request rate of 100 per second" when Client API rate limit is reached

Products

• VMware NSX

Issue/Introduction

• NSX has a default limit of 100 HTTP API requests per second. If this limit is exceeded, any additional API requests will start to fail.
• May receive error "Server is overloaded" when trying to access the NSX Manager UI when the API request limit of 100 is being exceeded.

Environment

• VMware NSX-T Data Center
• VMware NSX

Cause

When the limit of 100 requests per second is reached, the API endpoint will return the following error:

{ "error_code": 102, "error_message": "Client '######' exceeded request rate of 100 per second", "module_name": "common-services" },

"msg": "Status code was 429 and not [200]: HTTP Error 429: "

Note: ###### represents the username of the client who made the NSX API request.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

Since this issue is related to the API rate limit, please try one of the following options:

Option 1:

Reduce the number of API requests to stay under the 100 requests per second limit.

Option 2:

Retry the API request after some time. It should eventually succeed if the requests are kept below 100 per second.

Option 3:

Distribute the API requests by load balancing them, rather than targeting a single manager.
Using this method, you will incorporate a load balancer in the front of NSX Managers, where the 3 NSX Managers will be the pool members, represented by a load balanced IP address (not the NSX Manager's VIP!). 
Any solutions consuming NSX via APIs will be pointed to NSX through the load balanced IP address, instead of the NSX Manager's VIP (which is not load balanced).

In order to determine what services are sending the API request you can follow the steps below:

1. To find the Top API Requesters you can run the grep below.
   zgrep -Fh '/api/' /var/log/proxy/envoy_access_log.txt* 2> /dev/null | awk '{print $2}' | sort | uniq -c | sort -rn | head


2. This will sort the top frequency api request on a per second scale.
   zgrep -Fh '/api/' /var/log/proxy/envoy_access_log.txt* 2> /dev/null | awk '{print $1}' | cut -d '.' -f -1 | sort | uniq -c | sort -rn | head
   
   
3. Take the time stamp from the output of step 2 above and it will give you the requesters ip and how many API request were served that second.
   zgrep "2025-02-03T04:37:22" /var/log/proxy/envoy_access_log.txt* | awk '{print $2}' | sort | uniq -c | sort -rn
   
   
4. This will give the top API calls run in your environment at a given second point in time from the output or step2.
   zgrep -Fh '2025-02-03T04:37:22' /var/log/proxy/envoy_access_log.txt* 2> /dev/null | awk '{print $4,$5,$6}' | sort | uniq -c | sort -rn | head

Additional Information

To check the current client API request rate limit, follow these steps:

1. Establish an SSH session to the VIP-assigned NSX-T Manager.
2. Log in as the admin user.
3. Run the following command: get service http.

Also see:

• NSX API Guide - Designing API client code to work gracefully with rate limits
• KB article NSX-T API configuration for HTTP on NSX-T Manager.

 

For any additional details, please open a support ticket by following the steps outlined in the KB: Creating and managing Broadcom support cases.