NSX-T API configuration for HTTP on NSX-T Manager

Products

• VMware NSX

Issue/Introduction

Outlines settings for APIs in NSX-T and logging details

Environment

VMware NSX

Resolution

By default the rate limits for API calls to NSX-T manager are defined in the API guide:

In general they are split into 3 categories:

• per-client rate limit, in requests per second. If a client makes more requests than this limit in one second, the API server will refuse to service the API request and will return an HTTP 429 Too Many Requests Error. By default, this limit is 100 requests per second.
  
  
• per-client concurrency limit. This is the maximum number of outstanding requests that a client can have. For example, a client can open multiple connections to NSX and submit operations on each connection. When this limit is exceeded, the server returns a 503 Server Unavailable error to the client. By default, this limit is 40 concurrent requests.
  
  
• An overall maximum number of concurrent requests. This is the maximum number of API requests that can be in process on the server. If the server is at this limit, additional requests will be refused and the HTTP error 503 Service Unavailable will be returned to the client. By default, this limit is 199 concurrent requests.

Checking API rate limits

To confirm the API limits set on your environment, we can use the REST API or Cli (Command line).

REST API

Per node (the VIP rate will be determined by the cluster node it resides on):

GET /api/v1/cluster/{cluster-node-id}/node/services/http

 

To update values:

PUT /api/v1/cluster/{cluster-node-id}/node/services/http
PUT /api/v1/node/services/http

To restart the service:

POST /api/v1/cluster/{cluster-node-id}/node/services/http?action=start
POST /api/v1/node/services/http?action=start


Command line

As admin user on each manager type get service http:

Client API rate limit: 100 requests/sec
Client API concurrency limit: 40 connections
Global API concurrency limit: 199 connections

Note: these values should not be changed, unless requested by support and it is usually only a temporary measure. If the values are changed, the https service should be restarted for the new values to take effect restart service http/ui-service. The command used is set service http <rate to be changed>

The variables for these rates when  using set service http are as below: 

• client-api-concurrency-limit  Client API concurrency limit
• client-api-rate-limit         Client API rate limit (requests/sec)
• global-api-concurrency-limit  Global API concurrency limit
  
  

The UI http timeout session can be altered here also, using set service http session-timeout <timeout-value-in-seconds>

Then restarting the service restart service ui-service

More  details on using the command line, please refer to NSX CLI Guide

HTTP Logs

In NSX-T 4.0.1, the tomcat reverse-proxy was changed to envoy reverse-proxy. All API's flowing to the NSX-T manager and Global manager will pass through the reverse-proxy, before being forwarded to the backend services.

These logs can be found on the manager appliance under:

Pre 4.0.1

localhost_access_log.txt

2024-10-08T10:32:31.507Z ###.162.1.### - "GET /api/v1/transport-nodes/########-43f1-4d61-914c-############/network/interfaces/vmnic2/stats HTTP/1.1" 200 196 2891 2891

 

Post 4.0.1

envoy_access_log.txt

[2024-10-08T10:32:31.507Z] ###.162.1.### 172.##.#.### "GET" "/api/v1/transport-nodes/########-43f1-4d61-914c-############/network/interfaces/vmnic2/stats" "HTTP/1.1" 200 - 0 541 7 6 "###.162.1.###" "Apache-HttpClient/4.5.9 (Java/11.0.16)" "########-259c-4c8c-a747-###########" "172.##.#.###:443" "###.0.0.1:7440"

From both log entries above, much of the information is the same, from them the following can be interpreted:

• [2024-10-08T10:32:31.507Z] - Start time.
• ###.162.1.### - Remote IP address.
• 172.##.#.### - receiving IP address.
• GET - HTTP method
•  /api/v1/transport-nodes/########-43f1-4d61-914c-############/network/interfaces/vmnic2/stats - API sent.
• HTTP/1.1 - HTTP Protocol used.
• 200 - response code
• 0
• 541 - bytes sent.
• 7 - time taken to process the request in ms.
• 6 - time taken to commit the request in ms.

Additional Information

NSX API throws error message: "Client '######' exceeded request rate of 100 per second" when Client API rate limit is reached

Troubleshooting NSX API Calls

NSX-T API usage

NSX-T API Connection Limits for HTTP on NSX-T Manager