Failed to prep NSX hosts or unable to access embedded NSX-T UI when using vSphere vLCM

Products

• VMware NSX

Issue/Introduction

Informational
The issue and fix is at vCenter side and workaround can be applied to either vCenter or NSX.

Symptoms:
Issue 1: 
Failed to prepare NSX-T hosts in a vLCM cluster due to an error such as:

“Cannot determine whether NSX-T Data Center is enabled on this cluster. Go to NSX-T Data Center system and enable bidirectional trust.“

Even though the compute manager is up and trust has been enabled.

Issue 2:
Cannot access NSX-T UI from vCenter after deployment of an embedded OVA from vCenter

Environment

VMware NSX-T Data Center

Cause

Domains are case insensitive. This causes problems when use the domain name in case sensitive strings such as OIDC issuer url. 

The oidc issuer url is constructed using the getOIDCEntityID command which takes in the tenant (domain name) parameter.

When constructing the JWT, it always get the default tenant which is just a copy of the VC domain name such as  AG.local. (not lowercase)

Resolution

• This issue has been resolved in vCenter version 8.0U1 and newer versions.
  
  Workaround:

There are two workaround options available to the customers: 

Workaround to be implemented in vCenter :

Note:  Please take an offline snapshot of all VCs before proceeding.

• Using Jxplorer, modify the default tenant string

          Navigate to the dn:

           cn=Tenants,cn=IdentityManager,cn=Services,dc=AG,dc=local

• Modify the attribute vmwSTSDefaultTenant to ag.local (from AG.local).
• Restart VC services

Workaround to be implemented in NSX via API:

Change the issuer of oidc in NSX manually using API.

Note : If the compute manager's thumbprint updated, it will override the oidc changes and again same workaround has to be applied.

• Obtain the OIDC URI from NSX manager:

GET https://<nsxmgr>/api/v1/trust-management/oidc-uris

{

    "results": [

        {

            "oidc_uri": "https://<vcsa>/openidconnect/vsphere.local/.well-known/openid-configuration",

            "thumbprint": "<thumbprint-uuid>",

            "oidc_type": "vcenter",

            "scim_endpoints": [],

            "claim_map": [],

            "serviced_domains": [],

            "restrict_scim_search": false,

            "end_session_endpoint_uri": "https://<vcsa>/openidconnect/logout/vsphere.local",

            "issuer": "https://<vcsa>/openidconnect/vsphere.local",

            "jwks_uri": "https://<vcsa>/openidconnect/jwks/vsphere.local",

            "token_endpoint": "https://<vcsa>/openidconnect/token/vsphere.local",

             "claims_supported": [],

               "override_roles": [],

             "resource_type": "OidcEndPoint",

            "id": "<oidc-id>",

   "display_name": "<oidc-display-name>",

   "_create_time": 1675162497329,

   "_create_user": "<user-id>",

     "_last_modified_time": 1675162497329,

  "_last_modified_user": "<user-id>",

    "_system_owned": false,

   "_protection": "NOT_PROTECTED",

  "_revision": 0

        }

    ]

}

 

• In the vCenter Server log /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log, the correct URL for this OIDC can be found. 

[2023-11-13T13:04:45.849Z] [INFO ] http-nio-5090-exec-591       70052176 101815 ###### com.vmware.identity.websso.client.Message                         Incoming or outgoing SAML message.
 Message Type:AUTHN_REQUEST
 ID:<message_id>
 SessionIndex:null
 Message source:https://<vcsa>/ui/saml/websso/metadata
 Message destination:https://<vcsa>/websso/SAML2/SSO/vsphere.LOCAL ← URI will be used in the POST API command below. Only the FQDN and the vsphere.LOCAL components are needed.
 Message validation result (for incoming messages):null

 

• Delete the above OIDC URI obtained in the previous GET command.

DELETE https://<nsxmgr>/api/v1/trust-management/oidc-uris/<oidc-display-name>

• Create a new OIDC and make sure to match the SSO URI retrieved from the vCenter logs:

POST https://<nsxmgr>/api/v1/trust-management/oidc-uris/

Payload for this POST:

{

            "oidc_uri": "https://<vcsa>/openidconnect/vsphere.LOCAL/.well-known/openid-configuration",
            "thumbprint": "<>",
            "oidc_type": "vcenter",
            "scim_endpoints": [],
            "claim_map": [],
            "serviced_domains": [],
            "restrict_scim_search": false,
            "end_session_endpoint_uri": "https://<vcsa>/openidconnect/logout/vsphere.local",
            "issuer": "https://<vcsa>/openidconnect/vsphere.local",
            "jwks_uri": "https://<vcsa>/openidconnect/jwks/vsphere.local",
            "token_endpoint": "https://<vcsa>/openidconnect/token/vsphere.local",
            "claims_supported": [],
            "override_roles": [],
            "resource_type": "OidcEndPoint",
   }

 

Additional Information