Sporadic authentication failure on NSX Manager while using it with vIDM authentication

search cancel

Sporadic authentication failure on NSX Manager while using it with vIDM authentication

book

Article ID: 371332

calendar_today

Updated On:

Products

VMware NSX VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Authentication failures with 403 error codes are seen intermittently on NSX Manager while using it with vIDM authentication, especially for automated clients like vRA, vROPs, etc.
You may see an error similar to the following.

Error: [403] [The credentials were incorrect or the account specified has been locked.]

You see messages similar to the following in the /var/log/proxy/reverse-proxy.log file on NSX Manager:

2023-10-09T17:14:53.464Z ERROR https-#.#.#.#-443--exec-5 ExceptionUtils 2310041 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602) ~[spring-web-5.3.20.jar:5.3.20]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.initDiscoveryEndPoint(VidmTokenServices.java:234) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.init(VidmTokenServices.java:117) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.checkConfigChanged(VidmTokenServices.java:110) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.getTokenStore(VidmTokenServices.java:127) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.loadAuthentication(VidmTokenServices.java:259) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.OAuth2RestAuthenticationFilter.attemptAuthentication(OAuth2RestAuthenticationFilter.java:304) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.OAuth2RestAuthenticationFilter.doFilter(OAuth2RestAuthenticationFilter.java:201) ~[libreverse-proxy-compile.jar:?]
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at com.vmware.nsx.management.rp.security.SessionInvalidationFilter.doFilter(SessionInvalidationFilter.java:118) ~[libreverse-proxy-compile.jar:?]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) ~[spring-web-5.3.20.jar:5.3.20]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at com.vmware.nsx.management.rp.PreAuthenticationProxyFilter.doFilter(PreAuthenticationProxyFilter.java:61) ~[libreverse-proxy-compile.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar:5.3.20]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at com.vmware.nsx.management.rp.ApplicationInitializationFilter.doFilter(ApplicationInitializationFilter.java:115) ~[libreverse-proxy-compile.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at com.vmware.nsx.management.rp.ApiRateLimitingFilter.doFilter(ApiRateLimitingFilter.java:223) ~[libreverse-proxy-compile.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[catalina.jar:8.5.81]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[catalina.jar:8.5.81]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[catalina.jar:8.5.81]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[catalina.jar:8.5.81]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:367) ~[catalina.jar:8.5.81]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:639) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:882) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1691) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-util.jar:8.5.81]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-util.jar:8.5.81]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-util.jar:8.5.81]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776) ~[spring-web-5.3.20.jar:5.3.20]
        ... 60 more
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_362]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_362]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_362]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_362]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_362]
        at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_362]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776) ~[spring-web-5.3.20.jar:5.3.20]
        ... 60 more

2023-10-09T16:58:44.816Z ERROR https-#.#.#.#-443-exec-9 ExceptionUtils 5704 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>:443/#.#.#.#] failed: Connection timed out (Connection timed out)

2023-10-09T17:13:21.388Z ERROR https-#.#.#.#-443-exec-4 ExceptionUtils 2310041 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>:443/#.#.#.#] failed: Connection timed out (Connection timed out)

2023-10-09T17:14:53.464Z ERROR https-#.#.#.#-443-exec-5 ExceptionUtils 2310041 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>:443/#.#.#.#] failed: Connection timed out (Connection timed out)

mp="nsx-manager" level="WARNING" subcomp="http"] password grant flow authentication failed
2024-05-09T20:25:24.687Z  WARN Processing request 153db072-9186-42c3-94f8-af9319341c3d CustomOidcAuthorizationCodeAuthenticationProvider 95038 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] password grant flow authentication failed
2024-05-09T20:25:24.687Z  WARN Processing request b13c9537-####-####-####-da21fb1999b0 CustomOidcAuthorizationCodeAuthenticationProvider 95038 - [nsx@6876

You see messages similar to the following in the /var/log/proxy/localhost.log file on NSX Manager:

2024-06-04T14:05:38.419Z ERROR http-nio-127.0.0.1-6565-exec-316361 ExceptionUtils 140152 - [nsx@6876 comp="nsx-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)
2024-06-04T14:05:42.515Z ERROR http-nio-127.0.0.1-6565-exec-316363 ExceptionUtils 140152 - [nsx@6876 comp="nsx-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Impacted version:

Any version running kernel version 5.15.92 with the ip_blackhole feature still enabled.

The kernel version can be checked from the below command in the NSX manager Bash shell (root user):

root@NSX-Manager:~# uname -a

Run cat /proc/sys/kernel/grsecurity/ip_blackhole to see if the feature is enabled or not.

The above command returns 1 which means ip_blackhole feature is enabled.

Environment

VMware NSX-T Data Center 3.2.3
VMware NSX 4.0.x
VMware NSX 4.1.1.x
VMware NSX 4.1.2
VMware NSX 4.1.2.1

Cause

NSX-T Manager Appliances running kernel version 5.15.92 with the ip_blackhole feature enabled may reject certain traffic flows, causing TCP timeouts to occur.

Resolution

This issue is resolved in VMware NSX 3.2.4
This issue is resolved in VMware NSX 4.1.2.2
This issue is resolved in VMware NSX 4.2.0

Workaround

The workaround is to disable the ip_blackhole option of grsecurity, and this workaround does not have any functional impact on NSX.

Non-persistent workaround

Access all the 3 NSX Managers with the root user and apply the following command.

echo 0 > /proc/sys/kernel/grsecurity/ip_blackholeYou can validate this via the following command:

cat /proc/sys/kernel/grsecurity/ip_blackhole

The output should be 0

Note: This workaround will not remain effective if the Manager reboots. If the Manager is rebooted, the output cat /proc/sys/kernel/grsecurity/ip_blackhole will change back to 1. Which means the workaround needs to be applied again.

Persistent Workaround:

To make the workaround persistent after reboot, you must make the following change on all 3 NSX Managers:

Open the /etc/sysctl.d/60-nsx-common.conf file with a text editor

Add the following two lines to the end of the file:

# Disable kernel grsecurity IP blackholing
kernel.grsecurity.ip_blackhole = 0

Save and exit the file.

Reboot the NSX Manager node.

After reboot, check that the change has taken effect.

cat /proc/sys/kernel/grsecurity/ip_blackhole

If it is 0 , then it would be a confirmation that workaround has persisted after reboot of NSX Manager.

Additional Information

Impact

The workaround is to disable the ip_blackhole option of grsecurity, and this workaround does not have any functional impact on NSX.

Feedback

thumb_up Yes

thumb_down No