NSX manager UI using VIP is not loading: Suberror :15 Error Code: 101

Products

VMware NSX

Issue/Introduction

Symptoms:

You are unable to log into the NSX-T manager using the VIP address and are presented with the following error.

You can log into the managers directly, not using the VIP address.
The NSX Manager cluster will display as stable, when you run the get cluster status command as admin on the NSX-T manager cli.
All expected services will display as running, when you run the command get services as admin on the NSX-T manager cli.
In the NSX Manager log /var/log/proxy/localhost_access_log.txt you see the following 503 http response code:

2022-04-13T21:53:18.360Z 192.168.1.1 - "GET /j_spring_security_check HTTP/1.1" 503 987 0 0

In NSX Manager log/var/log/proxy/reverse-proxy.log we see the following:

2022-04-13T21:53:18.360Z INFO https-jsse-nio-4.0.2.111-443-exec-272 ApiRateLimitingFilter - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Number of global active requests exceeded allowed maximum of 199 3451 times in last 60 seconds

Environment

VMware NSX-T Data Center

Cause

The log entry exceeded allowed maximum, indicates the proxy server on the manager, the one which is leader for the VIP address, is busy with to many APIs in the queue. The http response code is expected under such instances and is meant to let API caller know it is busy and try again later.

Resolution

This issue is resolved in NSX version 3.2.1 or higher, whereby the manager will now issue an alert "server is overloaded" to allow the user to know the server is busy, due to too many API calls occurring.

Workaround:
Option 1
Identify the manager which holds the VIP address, login as admin and restart the proton service:

cli> restart service manager

This action will cause the VIP address to move to another manager. During the manager service restart, the manager will not be able to process request until it has completed restart.

This will temporarily resolve the issue, but if there are still too many API calls, the server will get overloaded again and be unable to service the requests, causing the issue to reoccur.
Review the servers making API calls to the manager(s) and reduce them.

Option 2
Identify the manager which holds the VIP address, login as admin and restart the server:

cli> reboot

This action will cause the VIP address to move to another manager. During the manager restart, the manager will not be able to process request until it has completed restart.

This will temporarily resolve the issue, but if there are still too many API calls, the server will get overloaded again and be unable to service the requests, causing the issue to reoccur.
Review the servers making API calls to the manager(s) and reduce them.

Note: To find the API leader, that is the manager which holds the VIP address, as admin on the NSX-T manager run get cluster status verbose.
Look for 'Leaders - SERVICE - api' this is the node UUID of the API leader.
Match this node UUID to the node listed above in 'Group Type: HTTPS - Members' to find the IP address of the manager which holds the leadership for API.