During Workload Management install, cluster selection returns "api request to NSX manager failed. Status code 403"
search cancel

During Workload Management install, cluster selection returns "api request to NSX manager failed. Status code 403"

book

Article ID: 321989

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere with Tanzu

Issue/Introduction

Symptoms:
  • When enabling Workload Management on vCenter connected to an NSX manager, after starting the configuration wizard and selecting NSX as the networking stack, there are no Clusters listed as COMPATIBLE.
  • Clicking on the INCOMPATIBLE list, user will see the expected cluster listed as INCOMPATIBLE. 
  • Incompatibility Reasons will show:
    • "Cluster domain-cX is missing compatible NSX-T VDS"
    • "Failed to list all distributed switches in vCenter <vCenter_ID>"
    • "API request to NSX Manager failed. Status code: 403, Status: 403"
  • /var/log/vmware/wcp/wcpsvc.log on vCenter will report errors like:
2023-01-10T05:41:57.233Z debug wcp [opID=60dcb77a] nsx InitCompatibleClusterSet incompatibleReasons: [{vcenter.wcp.nsx.list.dvs.error Failed to list all distributed switches in vCenter 7785d3db-5e3e-4110-8997-7b35f97084ac. [7785d3db-5e3e-4110-8997-7b35f97084ac] map[] 0xc000b9bca0} {vcenter.wcp.nsx.manager.http.error API request to NSX Manager failed. Status code: 403, Status: 403 . [403 403 ] map[] 0xc000b9bb90}]
  • /var/log/proton/nsxapi.log on NSX Manager reports errors like: 
2023-01-10T05:41:57.233Z ERROR http-nio-127.0.0.1-7440-exec-6 AuthenticationEntryPoint 4768 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] Not authenticated Full authentication is required to access this resource
org.springframework.security.authentication.InsufficientAuthenticationException: Full authentication is required to access this resource
        at org.springframework.security.web.access.ExceptionTranslationFilter.handleAccessDeniedException(ExceptionTranslationFilter.java:194) ~[?:?]
        at org.springframework.security.web.access.ExceptionTranslationFilter.handleSpringSecurityException(ExceptionTranslationFilter.java:173) ~[?:?]
  • /var/log/proxy/reverse-proxy.log on NSX Manager reports errors like:
2023-01-10T05:41:57.233Z INFO https-<VC_IP>-443-exec-9 VcTokenServices 5173 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] The access token has been successfully validated locally.
2023-01-10T05:41:57.233Z WARN https-<VC_IP>-443-exec-26 VcTokenServices 5173 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] Failed to get roles from JWT claims.
2023-01-10T05:41:57.233Z ERROR https-<VC_IP>-443-exec-26 VcAuthTokenInterceptingFilter 5173 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] Failed to Authenticate: User is not authorized to perform this operation on the application. Please contact the system administrator to get access.
  • Using DCLI, the following errors are reported:
# dcli +i
dcli> vcenter namespacemanagement distributedswitchcompatibility list --cluster domain-c8 --compatible false
Server error: com.vmware.vapi.std.errors.Error
Error messages:
    API request to NSX Manager failed. Status code: 403, Status: 403 .
    Failed to list all distributed switches in vCenter 7785d3db-5e3e-4110-8997-7b35f97084ac.


Environment

VMware vSphere 7.0 with Tanzu
VMware vSphere 8.0 with Tanzu

Cause

When vCenter connects to NSX-T manager in order to create Workload Management Supervisor Clusters, it delivers a JWT token to NSX to identify the user and privileges that will be used for object creation/management. This JWT token must contain the user ("wcp-<vCenterID>" user) and the roles the user can act as ("NsxAdministrators" in this case) in order for NSX to successfully return the requested information. If the user or privileges delivered in the JWT token are incorrect, NSX will return a 403 status code which indicates an authorization issue on the Client side (vCenter side). This problem can be caused by the following conditions:
 
  • vCenter is not added as Compute Manager in NSX-T or is Down/Unregistered. 
  • NSX Administrators group in vSphere is not attached to the NsxAdministrator role. This group should contain the "wcp-<vCenterID>" user (user will look like: "wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff"),
  • Or, the "wcp-<vCenterID>" user is not added as a member of the NSX Administrators group
  • Or, the "NsxAdministator" role has been deleted,
  • Or, the "wcp-<vCenterID>" Solution User has been added to a non-default group in vSphere with limited privileges.

Resolution

1. Ensure the vCenter is added to NSX-T manager as a Compute Manager and that it is Up and Registered:
  • Log into NSX Manager GUI, click on System -> Fabric -> Compute Managers
  • If the vCenter on which WCP is being enabled isn't added, add it.
  • If the vCenter on which WCP is being enabled is added but shows either Registration Status or Connection Status as Not Registered or Down, correct the problem.
  • It is also possible that the connection shows Up and Registered, but the certificate has changed or there is a stale entry. We can try to manually update the credentials with the following steps:
  • Click the Checkbox next to the Compute Manager
  • Select EDIT
  • To the right of "FQDN or IP Address", select EDIT
  • Re-enter the vCenter "[email protected]" username and password, then SAVE.

2. Ensure the "NSX Administrator" role is present on vCenter, if not, recreate it:
  • From vSphere Web Client, select Menu -> Administration
  • Under Access Control, select Roles
  • Search for NSX Administrator, if the role is not present, recreate it:
  • Connect to vCenter via SSH
  • Create a /usr/lib/vmware-wcp/py-modules/roles.py file and import config using the following command (ensure you copy the entire text from cat to the last EOF line):
# cat <<EOF >>/usr/lib/vmware-wcp/py-modules/roles.py
import featureState
from roles_groups_users_utils import Authz

if __name__ == "__main__":
    authz_patch = Authz()

authz_patch.ensure_roles_exist('/usr/lib/vmware-wcp/roles.xml')
EOF
  • Set the Python path with the following command:
# export PYTHONPATH="${PYTHONPATH}:/usr/lib/vmware/site-packages"
  • Run the python script to recreate roles using the following command:
# python /usr/lib/vmware-wcp/py-modules/roles.py
 
Example Output:

# python /usr/lib/vmware-wcp/py-modules/roles.py
2023-01-10T18:09:54.288Z  Role id 1004, name WorkloadStorageManagement - already exists
2023-01-10T18:09:54.293Z  Role id 1005, name vSphereKubernetesManager - already exists
2023-01-10T18:09:54.296Z  Role id 1007, name SupervisorServiceCluster - already exists
2023-01-10T18:09:54.299Z  Role id 1008, name SupervisorServiceRootFolder - already exists
2023-01-10T18:09:54.302Z  Role id 1009, name SupervisorServiceGlobal - already exists
2023-01-10T18:09:54.306Z  Role id 1021, name VMOperatorController - already exists
2023-01-10T18:09:54.308Z  Role id 1022, name VMOperatorControllerGlobal - already exists
2023-01-10T18:09:54.311Z  Role id 0, name VMServicesAdministrator - already exists
2023-01-10T18:09:54.314Z  Role id 0, name NsxAuditor - already exists
2023-01-10T18:09:54.316Z  Role id 0, name NsxViAdministrator - already exists
2023-01-10T18:09:54.325Z  Role id 0, name NsxAdministrator - added
 

3. Identify if the NsxAdministrators group exists and which users are added to it:

# /usr/lib/vmware-vmafd/bin/dir-cli group list --name NsxAdministrators
Enter password for [email protected]:
CN=wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff,CN=ServicePrincipals,DC=vsphere,DC=local
 
4. Create group if needed.
 
# /usr/lib/vmware-vmafd/bin/dir-cli ssogroup create --name NsxAdministrators

5. Add user to group if needed:
  • First, gather the "wcp-<vCenterID>" username we will add to the "NsxAdministrators" group:

# /usr/lib/vmware-vmafd/bin/dir-cli service list

Example Output:

# /usr/lib/vmware-vmafd/bin/dir-cli service list
Enter password for [email protected]:
1. machine-bd2c292a-8477-4fac-ac0e-88e68bc774ff
2. vsphere-webclient-bd2c292a-8477-4fac-ac0e-88e68bc774ff
3. vpxd-bd2c292a-8477-4fac-ac0e-88e68bc774ff
4. vpxd-extension-bd2c292a-8477-4fac-ac0e-88e68bc774ff
5. hvc-bd2c292a-8477-4fac-ac0e-88e68bc774ff
6. wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff
 
  • Next, add the "wcp-<vCenterID>" user to the NsxAdministrators group using the following command:
# /usr/lib/vmware-vmafd/bin/dir-cli group modify --name NsxAdministrators --add <solution_user_name>

Example Command:
 
# /usr/lib/vmware-vmafd/bin/dir-cli group modify --name NsxAdministrators --add wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff
 
Example Output:
 
# /usr/lib/vmware-vmafd/bin/dir-cli group modify --name NsxAdministrators --add wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff
Enter password for [email protected]:
Account [wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff] added to group [NsxAdministrators]
Group member [wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff] added successfully
 
6. Ensure the "NsxAdministrators" group is added to the "NSX Administrator" Role in vSphere Web Client:
  • From vSphere Web Client, select Menu -> Administration
  • Under Access Control, select Global Permissions
  • Search for VSPHERE.LOCAL\NsxAdministrators, ensure that it has role NSX Administrator assigned. If you don't see the user present, add it:
    • Select ADD
    • In "Domain" field, select vsphere.local (or the local vmdir domain)
    • In the "User/Group" field, select "NsxAdministrators"   - ENSURE THERE ARE NO SPACES BEFORE OR AFTER THE NAME
    • In the "Role" field, select "NSX Administrator"
    • Check the "Propagate to childres" option, then click OK.

7. Ensure the wcp-<vCenterID> user is not added to any groups it shouldn't be added to:
  • User should be added to only the following groups:
    • SolutionUsers
    • ActAsUsers
    • NsxAdministrators
 


Additional Information

Impact/Risks:
This error will prevent WCP Workload Management cluster instantiation.

Powered by Wolken Software