NSX Global Manager UI is generating error "Server is overloaded (Error code: 98)"

Products

• VMware NSX

Issue/Introduction

• NSX global manager UI stops working with following error -

HTTP failure with Error: Server is overloaded (Error code: 98)","error_code":"98","module_name":"common-service","error_data":{"status":503}}}.

• Login to the NSX Global Manager as user admin and switch to the root user account by running the command st en

  Navigate to the location /var/log/gmanager/gmanager-ui.log shows log similar to following -
  
  INFO http-nio-127.0.0.1-64440-exec-42 UI_LOG #### - [nsx@6876 comp="global-manager" level="INFO" reqId="####-####-####-####-####" subcomp="global-manager" username="####@####"] {"user":"####@####","message":"Api Errors->","messageData":{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":503,"statusText":"Service Unavailable","url":"https://<nsx-global-manager>/api/v1/upgrade/available-releases?source=notification","ok":false,"name":"HttpErrorResponse","message":"Http failure response for https://<nsx-global-manager>/api/v1/upgrade/available-releases?source=notification: 503 Service Unavailable","error":{"error_code":"98","module_name":"common-service","error_message":"Error: Server is overloaded (Error code: 98)","error_data":{"status":503}}},"level":"Error","browser":"####","time":"####","location":""}
  
  
• Further, check the log /var/log/proxy/envoy.log which shows concurrency limit reached.

[info][filter] [nsx_envoy_filters/source/http/peruserpernodeapilimit/per_user_per_node_api_limit.cc:41] concurrency limit reached 40 for username_or_ip Qj####==

Environment

VMware NSX

Cause

The issue happens if per_user_per_node_api_limit has maximum limit.

Resolution

• Identify the client exceeding limit and take appropriate steps to ensure the API calls from that client do not the threshold limit.
  
  
• Check configured client API concurrency limit using following API or CLI -
  
  CLI :
  nsx-mngr> get service http

Service name:                     http
Service state:                    running
Logging level:                    info
Session timeout:                  1800
Connection timeout:               30
Client API rate limit:            100 requests/sec
Client API concurrency limit:     40 connections
Global API concurrency limit:     199 connections
Redirect host:                    (not configured)
Basic authentication:             enabled
Cookie-based authentication:      enabled
  
  API :

GET https://<nsx-mgr>/api/v1/cluster/api-service
or, 
curl -k -u admin -X GET "https://<nsx-mgr>/api/v1/cluster/api-service"

• This is the maximum number of outstanding requests that a client can have.
• For example, a client can open multiple connections to NSX and submit operations on each connection. When this limit is exceeded, the server returns a 503 Server Unavailable error to the client. By default, this limit is 40 concurrent requests.
• If more connections are required, then update the configured value to desired limit. 
  
  Note : Do not modify or increase the default limits unless a specific use case justifies the change. Increasing this limit will masks underlying issues, instead it is recommended to take appropriate steps to limit API calls from the offending client.
  
  Steps to update client API concurrency limit -
  
  CLI : 
• Login to the NSX manager as user admin and run the command below
  
  <nsx-manager> set service http client-api-concurrency-limit
  <http-client-api-concurrency-limit>  HTTP API per-client concurrency limit value in the range of 0 - 9223372036854775807
  
  API :

https://developer.broadcom.com/xapis/nsx-t-data-center-rest-api/4.2.1/method_UpdateApiServiceConfig.html 

PUT https://<nsx-mgr>/api/v1/cluster/api-service
{
  "global_api_concurrency_limit": 199,
  "client_api_rate_limit": 100,
  "client_api_concurrency_limit": 40, <-- [Set required value here]
  "connection_timeout": 30,
  "redirect_host": "",
  "cipher_suites": [
    {"enabled": true, "name": "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"},
    {"enabled": true, "name": "TLS_RSA_WITH_AES_256_GCM_SHA384"},
    {"enabled": true, "name": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"},
    {"enabled": true, "name": "TLS_RSA_WITH_AES_128_GCM_SHA256"}
    {"enabled": true, "name": "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384}",
    {"enabled": true, "name": "TLS_RSA_WITH_AES_256_CBC_SHA256"},
    {"enabled": true, "name": "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"},
    {"enabled": true, "name": "TLS_RSA_WITH_AES_256_CBC_SHA"},
    {"enabled": true, "name": "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"},
    {"enabled": true, "name": "TLS_RSA_WITH_AES_128_CBC_SHA256"},
    {"enabled": false, "name": "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"},
    {"enabled": false, "name": "TLS_RSA_WITH_AES_128_CBC_SHA"}
  ],
  "protocol_versions": [
    {"enabled": true, "name": "TLSv1.1"},
    {"enabled": false, "name": "TLSv1.2"}
  ]
}

or, if you are using curl please use the following command -
curl -k -u admin -H "Content-Type: application/json" -X PUT "https://<nsx-mgr>/api/v1/cluster/api-service" -d @api-service.json

Here api-service.json is the name of file containing body.

Additional Information

Please refer to following KBs for more information on API configuration and API access logs.
NSX-T API configuration for HTTP on NSX-T Manager
'Manager CPU usage very high' alarm is generated for NSX manager VIP node