Intermittent HTTP 403 Forbidden Errors During Long-Running NSX Manager API Calls for vIDM Users

Products

• VMware NSX

Issue/Introduction

• When using VMware Identity Manager (vIDM) accounts to authenticate, users experience intermittent HTTP 403 Forbidden errors when executing long-running API calls to the NSX Manager.
• During these events, API requests fail, and the reverse-proxy logs on the NSX Manager display errors such as "Authentication validation failed", "The credentials were incorrect or the account specified has been locked" and "OAuth2AccessDeniedException: Access token denied."
  Packet captures confirm there is no loss of network connectivity between the NSX Manager and the VMware Identity Manager (vIDM) during the time of the issue.
• The following Envoy access log snippets demonstrate the 403 Forbidden errors during the API execution:

[] "GET" "/api/v1/ns-groups" "HTTP/1.1" 403 UAEX 0 141 1039 - "" ""
[] "GET" "/api/v1/ns-groups?cursor=..." "HTTP/1.1" 403 UAEX 0 141 1052 - "" ""

• The following reverse-proxy log snippets detail the exact authentication and token rejection exceptions:

INFO Processing request OAuth2AuthenticationProvider 74473 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Failed to use SAMAccountName, attempting UserPrincipleName: Invalid credentials ERROR Processing request SessionInvalidationFilter 74473 - [nsx@6876 comp="nsx-manager" errorCode="MP1" level="ERROR" subcomp="http"] Authentication validation failed. ERROR Processing request NsxRestAuthenticationEntryPoint 74473 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] The credentials were incorrect or the account specified has been locked. ... org.springframework.security.authentication.BadCredentialsException: Invalid credentials ... Caused by: org.springframework.security.oauth2.client.resource.OAuth2AccessDeniedException: Access token denied.

 

• vIDM horizon.log:
  

  UserPasswordValidationService - Validating password for user <user_name>..

   

• vIDM connector.log:
  

  ConnectorAuthBrokerRestController - Failed to login. Internal server error.
  java.lang.IllegalArgumentException: password is required

Environment

VMware NSX
VMware Identity Manager

Cause

This issue occurs due to a known defect within the NSX Manager session management mechanism (SessionInvalidationFilter).

When API clients or automated tools (like Terraform) utilize the Password Grant flow (via /api/session/create), NSX initiates a background security validation check every 5 minutes (300 seconds). During this check, NSX erroneously transmits an empty password parameter (password=) to VMware Identity Manager (vIDM). The vIDM connector rejects the empty string, causing the background re-validation to fail, which prematurely terminates the active session and returns intermittent HTTP 403 Forbidden errors.

Resolution

This issue is a known defect and is formally fixed in NSX 9.x versions.

Workaround:

If you are on an affected branch (such as NSX 4.2.x) and cannot immediately upgrade, you can mitigate the frequency of these errors by increasing the background session validation timeout interval on each NSX Manager appliance:

1. Log into the NSX Manager appliance shell as root.

2. Open the HTTP properties configuration file: /config/http/http.properties

3. Add or modify the following line to increase the interval (the default is 300 seconds / 5 minutes). Set it to a value that comfortably accommodates your longest-running pipeline iterations: authentication.session.validationTimeout = <Substantially_Higher_Value_In_Seconds>

4. Restart the reverse-proxy service for the changes to take effect: systemctl restart proxy

Note: Modifying VIDM_USER_ACCESS_TOKEN_TTL via script only affects UI browser sessions and will not resolve this API behavior.

Additional Information

Reference articles:
LDAP users got logged out of NSX 9.x within 5 minutes