NSX APIs throws error message: "Client '######' exceeded request rate of 100 per second" when Client API rate limit is reached

Products

• VMware NSX

Issue/Introduction

NSX-T has a default limit of 100 HTTP API requests per second. If this limit is exceeded, any additional API requests will start to fail.

Environment

VMware NSX-T Data Center

VMware NSX

Cause

When the limit of 100 requests per second is reached, the API endpoint will return the following error:

{ "error_code": 102, "error_message": "Client '######' exceeded request rate of 100 per second", "module_name": "common-services" },

"msg": "Status code was 429 and not [200]: HTTP Error 429: "

Note: ###### represents the username of the client who made the NSX API request.

Resolution

Since this issue is related to the API rate limit, please try one of the following options:

Option 1:
Reduce the number of API requests to stay under the 100 requests per second limit.

Option 2:
Retry the API request after some time. It should eventually succeed if the requests are kept below 100 per second.

Option 3:
Distribute the API requests by load balancing them, rather than targeting a single manager.

In order to determine what services are sending the api request you can follow the steps below.

          1. To find the Top API Requesters you can run the grep below.
       zgrep -Fh '/api/' /var/log/proxy/envoy_access_log.txt* 2> /dev/null | awk '{print $2}' | sort | uniq -c | sort -rn | head

          2. This will sort the top frequency api request on a per second scale.
       zgrep -Fh '/api/' /var/log/proxy/envoy_access_log.txt* 2> /dev/null | awk '{print $1}' | cut -d '.' -f -1 | sort | uniq -c | sort -rn | head

          3. Take the time stamp from the output of step 2 above and it will give you the requesters ip and how many API request were served that second.
       zgrep "2025-02-03T04:37:22" /var/log/proxy/envoy_access_log.txt* | awk '{print $2}' | sort | uniq -c | sort -rn

          4. This will give the top api calls run in your environment at a given second point in time from the output or step2.
       zgrep -Fh '2025-02-03T04:37:22' /var/log/proxy/envoy_access_log.txt* 2> /dev/null | awk '{print $4,$5,$6}' | sort | uniq -c | sort -rn | head

Additional Information

To check the current client API request rate limit, follow these steps:

1. Establish an SSH session to the VIP-assigned NSX-T Manager.
2. Log in as the admin user.
3. Run the following command: get service http.

For any additional details, please open a support ticket by following the steps outlined in the KB: Creating and managing Broadcom support cases