REST API to VMware NSX Manager fails while using it with vIDM authentication

Products

VMware NSX

Issue/Introduction

VMware NSX deployed with vIDM authentication.
VMware vRealise Automation may also be in use.
The below entries can be encountered in var/log/proxy/reverse-proxy.log 2024-08-20T14:51:51.460Z NSX ExceptionUtils 804913 - [nsx@6876 comp="nsx-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<vIDM_hostname>/SAAS/auth/.well-known/openid-configuration": Connect to <vIDM_hostname>:443 [<vIDM_hostname>/##.##.##.##] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <vIDM_hostname>:443 [<vIDM_hostname>/##.##.##.##] failed: Connection timed out (Connection timed out)
API calls with a 403 response may be visible in var/log/proxy/localhost_access_log.txt
2024-08-20T14:51:51.950Z ##.##.##.## - "GET /api/v1/####### HTTP/1.1" 403 218 30 29
Entries similar to the below can be encountered in var/log/proxy/reverse-proxy.log
2024-08-20T14:51:51.949Z WARN NSX CustomOidcAuthorizationCodeAuthenticationProvider 73303 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] password grant flow authentication failed
2024-08-20T14:51:51.949Z INFO NSX NsxBasicAuthenticationFilter 73303 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Invalid credentials
2024-08-20T14:51:51.949Z ERROR NSX NsxRestAuthenticationEntryPoint 73303 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] The credentials were incorrect or the account specified has been locked.

Environment

VMware NSX 4.x

VMware NSX-T Data Center 3.x

Cause

This issue is encountered due to overloading the vIDM server but there is an issue that can contribute to overloading the server.

In versions of VMware NSX prior to 3.2.3.2 the behavior of the product could see the source port selection number vary widely, even for connections to the same host, and so one could observe several port number reused there. If the client reused a port number while the server was still attempting to close the old connection using that port number and the various ID numbers happened to pass validation, this would cause unexpected termination of the new connection.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.3.2, VMware NSX 4.1.2.2 and later versions available at Broadcom downloads.

Upgrading would likely reduce the instances that this issue is encountered, but it is possible that a review of the configuration for vIDM and vRA may be necessary as this can be encountered due to a high volume of authentication events occurring between vIDM and the VMware NSX manager.

Workaround

To disable the blackhole feature which randomizes the client ports, carry out the below steps on the NSX Managers.

On a VMware NSX Manager execute the below commands:

systemctl stop proxy
In /etc/init.d/proxy, add the echo command to the prestart() function:
prestart() {
echo 0 > /proc/sys/kernel/grsecurity/ip_blackhole
systemctl daemon-reload
systemctl start proxy
Check with /proc/sys/kernel/grsecurity/ip_blackhole