Sporadic authentication failure on NSX Manager while using it with vIDM authentication

search cancel

Sporadic authentication failure on NSX Manager while using it with vIDM authentication

book

Article ID: 371332

calendar_today

Updated On:

Products

VMware NSX VMware Aria Suite

Issue/Introduction

Authentication failures with 403 error codes are seen intermittently on NSX Manager while using it with vIDM authentication, especially for automated clients like vRA, vROPs, etc.
You may see an error similar to the following

Error: [403] [The credentials were incorrect or the account specified has been locked.]

You see messages similar to the following in the /var/log/proxy/reverse-proxy.log file on NSX Manager:

2023-10-09T17:14:53.464Z ERROR https-#.#.#.#-443--exec-5 ExceptionUtils 2310041 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602) ~[spring-web-5.3.20.jar:5.3.20]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.initDiscoveryEndPoint(VidmTokenServices.java:234) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.init(VidmTokenServices.java:117) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.checkConfigChanged(VidmTokenServices.java:110) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.getTokenStore(VidmTokenServices.java:127) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.VidmTokenServices.loadAuthentication(VidmTokenServices.java:259) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.OAuth2RestAuthenticationFilter.attemptAuthentication(OAuth2RestAuthenticationFilter.java:304) ~[libreverse-proxy-compile.jar:?]
        at com.vmware.nsx.management.rp.security.oauth2.OAuth2RestAuthenticationFilter.doFilter(OAuth2RestAuthenticationFilter.java:201) ~[libreverse-proxy-compile.jar:?]
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at com.vmware.nsx.management.rp.security.SessionInvalidationFilter.doFilter(SessionInvalidationFilter.java:118) ~[libreverse-proxy-compile.jar:?]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) ~[spring-security-web-5.5.8.jar:5.5.8]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) ~[spring-web-5.3.20.jar:5.3.20]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at com.vmware.nsx.management.rp.PreAuthenticationProxyFilter.doFilter(PreAuthenticationProxyFilter.java:61) ~[libreverse-proxy-compile.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar:5.3.20]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at com.vmware.nsx.management.rp.ApplicationInitializationFilter.doFilter(ApplicationInitializationFilter.java:115) ~[libreverse-proxy-compile.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at com.vmware.nsx.management.rp.ApiRateLimitingFilter.doFilter(ApiRateLimitingFilter.java:223) ~[libreverse-proxy-compile.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[catalina.jar:8.5.81]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[catalina.jar:8.5.81]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[catalina.jar:8.5.81]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698) ~[catalina.jar:8.5.81]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[catalina.jar:8.5.81]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:367) ~[catalina.jar:8.5.81]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:639) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:882) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1691) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-coyote.jar:8.5.81]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-util.jar:8.5.81]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-util.jar:8.5.81]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-util.jar:8.5.81]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776) ~[spring-web-5.3.20.jar:5.3.20]
        ... 60 more
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_362]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_362]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_362]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_362]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_362]
        at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_362]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-5.3.20.jar:5.3.20]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776) ~[spring-web-5.3.20.jar:5.3.20]
        ... 60 more

2023-10-09T16:58:44.816Z ERROR https-#.#.#.#-443-exec-9 ExceptionUtils 5704 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>:443/#.#.#.#] failed: Connection timed out (Connection timed out)

2023-10-09T17:13:21.388Z ERROR https-#.#.#.#-443-exec-4 ExceptionUtils 2310041 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>:443/#.#.#.#] failed: Connection timed out (Connection timed out)

2023-10-09T17:14:53.464Z ERROR https-#.#.#.#-443-exec-5 ExceptionUtils 2310041 - [nsx@6876 comp="global-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>:443/#.#.#.#] failed: Connection timed out (Connection timed out)

mp="nsx-manager" level="WARNING" subcomp="http"] password grant flow authentication failed
2024-05-09T20:25:24.687Z  WARN Processing request 153db072-9186-42c3-94f8-af9319341c3d CustomOidcAuthorizationCodeAuthenticationProvider 95038 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] password grant flow authentication failed
2024-05-09T20:25:24.687Z  WARN Processing request b13c9537-####-####-####-da21fb1999b0 CustomOidcAuthorizationCodeAuthenticationProvider 95038 - [nsx@6876

You see messages similar to the following in the /var/log/proxy/localhost.log file on NSX Manager:

2024-06-04T14:05:38.419Z ERROR http-nio-127.0.0.1-6565-exec-316361 ExceptionUtils 140152 - [nsx@6876 comp="nsx-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/10.118.196.10] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)
2024-06-04T14:05:42.515Z ERROR http-nio-127.0.0.1-6565-exec-316363 ExceptionUtils 140152 - [nsx@6876 comp="nsx-manager" errorCode="MP98" level="ERROR" subcomp="http"] Uncaught exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "<FQDN>/SAAS/auth/.well-known/openid-configuration": Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to <FQDN>:443 [<FQDN>/#.#.#.#] failed: Connection timed out (Connection timed out)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center3.2.3
VMware NSX 4.0.x
VMware NSX 4.1.1.x
VMware NSX 4.1.2
VMware NSX 4.1.2.1

Cause

This issue is caused by an issue with the ip_blackhole option of grsecurity on NSX Manager

Resolution

This issue is resolved in VMware NSX 3.2.4
This issue is resolved in VMware NSX 4.1.2.2
This issue is resolved in VMware NSX 4.2.0

Workaround

The workaround is to disable ip_blackhole option of grsecurity.

Non-persistent workaround

Access all the 3 NSX Managers with root user and apply the following command

echo 0 > /proc/sys/kernel/grsecurity/ip_blackholeYou can validate that this via the following command:

cat /proc/sys/kernel/grsecurity/ip_blackhole

The output should be 0

Note: This workaround will not remain effective if the Manager reboots. If the Manager is rebooted, the output of cat /proc/sys/kernel/grsecurity/ip_blackhole will change back to 1. Which means the workaround needs to be applied again.

Persistent Workaround:

To make the workaround persistent after reboot, you must make the following change on all 3 NSX Managers:
- Open the /etc/sysctl.d/60-nsx-common.conf file with a text editor
- Add the following two lines to the end of the file:
  
  # Disable kernel grsecurity IP blackholing
  kernel.grsecurity.ip_blackhole = 0
- Save and exit the file
- Reboot the NSX Manager node
- After reboot, check that the change has taken effect
  
  cat /proc/sys/kernel/grsecurity/ip_blackhole
  
  If it is 0 , then it would be a confirmation that workaround has persisted after reboot of NSX Manager.

Additional Information

Impact

Automated clients like vRA, vROPs vRA do not work as expected due to 403 error codes returned from the NSX Manager.
vRA deployments may fail.

Feedback

thumb_up Yes

thumb_down No