NSX manager UI using VIP is not loading: Suberror :15 Error Code: 101
Article ID: 322459
Updated On:
Products
VMware NSX Networking
Issue/Introduction
Symptoms:
- You are unable to log into the NSX-T manager using the VIP address and are presented with the following error.
- You can log into the managers directly, not using the VIP address.
- The NSX Manager cluster will display as stable, when you run the get cluster status command as admin on the NSX-T manager cli.
- All expected services will display as running, when you run the command get services as admin on the NSX-T manager cli.
- In the NSX Manager log /var/log/proxy/localhost_access_log.txt you see the following 503 http response code:
2022-04-13T21:53:18.360Z 192.168.1.1 - "GET /j_spring_security_check HTTP/1.1" 503 987 0 0
- In NSX Manager log/var/log/proxy/reverse-proxy.log we see the following:
2022-04-13T21:53:18.360Z INFO https-jsse-nio-4.0.2.111-443-exec-272 ApiRateLimitingFilter - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Number of global active requests exceeded allowed maximum of 199 3451 times in last 60 seconds
Environment
VMware NSX-T Data Center
Cause
The log entry exceeded allowed maximum, indicates the proxy server on the manager, the one which is leader for the VIP address, is busy with to many APIs in the queue. The http response code is expected under such instances and is meant to let API caller know it is busy and try again later.
Resolution
This issue is resolved in NSX version 3.2.1 available at VMware downloads, whereby the manager will now issue an alert "server is overloaded" to allow the user to know the server is busy, due to to many API calls occurring.
Workaround:
Option 1
Identify the manager which holds the VIP address, login as admin and restart the proton service:
This will temporarily resolve the issue, but if there are still to many API calls, the server will get overloaded again and be unable to service the requests, causing the issue to reoccur.
Review the servers making API calls to the manager(s) and reduce them.
Option 2
Identify the manager which holds the VIP address, login as admin and restart the server:
This will temporarily resolve the issue, but if there are still to many API calls, the server will get overloaded again and be unable to service the requests, causing the issue to reoccur.
Review the servers making API calls to the manager(s) and reduce them.
Note: To find the API leader, that is the manager which holds the VIP address, as admin on the NSX-T manager run get cluster status verbose.
Look for 'Leaders - SERVICE - api' this is the node UUID of the API leader..
Match this node UUID to the node listed above in 'Group Type: HTTPS - Members' to find the IP address of the manager which holds the leadership for API.
Workaround:
Option 1
Identify the manager which holds the VIP address, login as admin and restart the proton service:
cli> restart service manager
This action will cause the VIP address to move to another manager. During the manager service restart, the manager will not be able to process request until it has completed restart.This will temporarily resolve the issue, but if there are still to many API calls, the server will get overloaded again and be unable to service the requests, causing the issue to reoccur.
Review the servers making API calls to the manager(s) and reduce them.
Option 2
Identify the manager which holds the VIP address, login as admin and restart the server:
cli> reboot
This action will cause the VIP address to move to another manager. During the manager restart, the manager will not be able to process request until it has completed restart.This will temporarily resolve the issue, but if there are still to many API calls, the server will get overloaded again and be unable to service the requests, causing the issue to reoccur.
Review the servers making API calls to the manager(s) and reduce them.
Note: To find the API leader, that is the manager which holds the VIP address, as admin on the NSX-T manager run get cluster status verbose.
Look for 'Leaders - SERVICE - api' this is the node UUID of the API leader..
Match this node UUID to the node listed above in 'Group Type: HTTPS - Members' to find the IP address of the manager which holds the leadership for API.
Feedback
Yes
No
Powered by
