“Cannot determine whether NSX-T Data Center is enabled on this cluster. Go to NSX-T Data Center system and enable bidirectional trust.“ Error when trying to install NSX VIBs on vLCM enabled cluster

Products

VMware NSX

Issue/Introduction

The NSX-T cluster is configured to use vLCM (vSphere Lifecycle Manager) on vCenter.
In NSX-T the compute manager has been correctly registered with Trust enabled and Service account created.
You get the following error when trying to prepare (Install NSX VIBs), on the vLCM enabled cluster:

“Cannot determine whether NSX-T Data Center is enabled on this cluster. Go to NSX-T Data Center system and enable bidirectional trust.“

In the vCenter logs /var/log/vmware/vum-server/vmware-vum-server.log we see following log entries:

error vmware-vum-server[10700] [Originator@6876 sub=vmomi.soapStub[76]] Initial service state request failed, disabling pings; /sso-adminserver/sdk/vsphere.local, <last binding: <<TCP '127.0.0.1 : 36042'>, <TCP '127.0.0.1 : 443'>>>, HTTP Status:405 'Method Not Allowed'
info vmware-vum-server[25537] [Originator@6876 sub=SsoClient] Successfully acquired token:
SamlToken [subject={Name: vpxd-extension-########-####-####-####-############; Domain:vSphere.local},
groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: ServiceProviderUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vSphere.local}],
delegationChain=[], startTime=<date/time>, expirationTime=<date/time>, renewable=false, delegable=false, isSolution=true,confirmationType=1]
...
info vmware-vum-server[25537] [Originator@6876 sub=EHP] Found cached JWT
error vmware-vum-server[25537] [Originator@6876 sub=EHP] Response from ##.##.#.#/api/v1/node/services/install-upgrade: HTTP Status:403 'Forbidden'
error vmware-vum-server[25537] [Originator@6876 sub=EHP] Failed to call NSX-T/api/v1/node/services/install-upgrade
error vmware-vum-server[25537] [Originator@6876 sub=EHP] Caught exception while finding Nsxt Upgrade Coordinator: Failed to call NSX-T/api/v1/node/services/install-upgrade

API call GET /api/v1/trust-management/oidc-uris to NSX-T manager shows the lists of OIDC endpoints:

"results" : [ {
"oidc_uri" : "https://<VC-FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration",
"thumbprint" : "<UUID>",
"oidc_type" : "vcenter",
"issuer" : "https://<VC-FQDN>/openidconnect/vsphere.local",
"jwks_uri" : "https://<VC-FQDN>/openidconnect/jwks/vsphere.local",
"token_endpoint" : "https://<VC-FQDN>/openidconnect/token/vsphere.local",
"claims_supported" : [ ],
"override_roles" : [ ],
"resource_type" : "OidcEndPoint",

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.x
VMware NSX-T Data Center 3.x

Cause

This issue occurs when the domain name registered as the OIDC endpoint (vCenter) case sensitivity is different from the token issuer in vCenter.
From the log entry we see vSphere.local for the vpxd-extension and Everyone group.
OIDC issuer is registered with all lowercase entry vsphere.local, as can be seen in the issue line of the API response.

Resolution

This issue has been resolved in vCenter version 8.0U1 and newer versions.

Workaround to be implemented in NSX via API:

Change the issuer of oidc in NSX manually using API, please make sure and do an NSX backup before proceeding.

Note: If the compute manager's thumbprint is updated in NSX, after applying this workaround, it will override the oidc changes here and the workaround will need to be reapplied.

Obtain the OIDC URI from NSX manager:

GET https://<nsxmgr>/api/v1/trust-management/oidc-uris

              {"results": [{
"oidc_uri": "https://<vcsa>/openidconnect/vsphere.local/.well-known/openid-configuration",
"thumbprint": "<thumbprint-uuid>",
"oidc_type": "vcenter",
"scim_endpoints": [],
"claim_map": [],
"serviced_domains": [],
"restrict_scim_search": false,
"end_session_endpoint_uri": "https://<vcsa>/openidconnect/logout/vsphere.local",
"issuer": "https://<vcsa>/openidconnect/vsphere.local",
"jwks_uri": "https://<vcsa>/openidconnect/jwks/vsphere.local",
"token_endpoint": "https://<vcsa>/openidconnect/token/vsphere.local",
   "claims_supported": [],
   "override_roles": [],
    "resource_type": "OidcEndPoint",
"id": "<oidc-id>",
"display_name": "<oidc-display-name>",
"_create_time": 1675162497329,
"_create_user": "<user-id>",
                     "_last_modified_time": 1675162497329,
"_last_modified_user": "<user-id>",
"_system_owned": false,
  "_protection": "NOT_PROTECTED",
                     "_revision": 0
}]}

In the vCenter Server log /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log, the correct URL for this OIDC can be found.

[INFO ] http-nio-5090-exec-591 70052176 101815 ###### com.vmware.identity.websso.client.Message Incoming or outgoing SAML message.
Message Type:AUTHN_REQUEST
ID:<message_id>
SessionIndex:null
Message source:https://<vcsa>/ui/saml/websso/metadata
Message destination:https://<vcsa>/websso/SAML2/SSO/vSphere.local ← URI will be used in the POST API command below. Only the FQDN and the vSphere.local components are needed.
Message validation result (for incoming messages):null

Delete the above OIDC URI obtained in the previous GET command.

DELETE https://<nsxmgr>/api/v1/trust-management/oidc-uris/<oidc-display-name>

Create a new OIDC and make sure to match the SSO URI retrieved from the vCenter logs:

POST https://<nsxmgr>/api/v1/trust-management/oidc-uris/

Payload for this POST:

{

"oidc_uri": "https://<vcsa>/openidconnect/vSphere.local/.well-known/openid-configuration",
"thumbprint": "<>",
"oidc_type": "vcenter",
"scim_endpoints": [],
"claim_map": [],
"serviced_domains": [],
"restrict_scim_search": false,
"end_session_endpoint_uri": "https://<vcsa>/openidconnect/logout/vsphere.local",
"issuer": "https://<vcsa>/openidconnect/vsphere.local",
"jwks_uri": "https://<vcsa>/openidconnect/jwks/vsphere.local",
"token_endpoint": "https://<vcsa>/openidconnect/token/vsphere.local",
"claims_supported": [],
"override_roles": [],
"resource_type": "OidcEndPoint",
}

Additional Information

If this KB did not help resolve your issue, you can review the following KB for further troubleshooting steps Troubleshooting NSX Installation Operations.

If you are contacting Broadcom support about this issue, please provide the following:

NSX Manager and vCenter support bundles
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support