NSX-T fails to install VIBs on vLCM enabled cluster

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• The NSX-T cluster is configured to use vLCM (vSphere Lifecycle Manager) on vCenter.
• On NSX-T the compute manager has been correctly registered with Trust enabled and Service account created.
• In the vCenter logs /var/log/vmware/vum-server/vmware-vum-server.log we see following log entries:

2022-11-22T18:23:12.477-06:00 info vmware-vum-server[25537] [Originator@6876 sub=EHP] Acquiring SAML token with extension certificate...
2022-11-22T18:23:12.522-06:00 error vmware-vum-server[10700] [Originator@6876 sub=vmomi.soapStub[76]] Initial service state request failed, disabling pings; /sso-adminserver/sdk/vsphere.local, <last binding: <<TCP '127.0.0.1 : 36042'>, <TCP '127.0.0.1 : 443'>>>, HTTP Status:405 'Method Not Allowed'
2022-11-22T18:23:12.547-06:00 info vmware-vum-server[25537] [Originator@6876 sub=SsoClient] Successfully acquired token: 
SamlToken [subject={Name: vpxd-extension-525ea8dc-48be-4df7-8503-4567831850e8; Domain:vSphere.local}, 
groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: ServiceProviderUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vSphere.local}], 
delegationChain=[], startTime=2022-11-23 00:23:12.527, expirationTime=2022-11-23 01:23:12.527, renewable=false, delegable=false, isSolution=true,confirmationType=1]
...
2022-11-22T18:23:12.547-06:00 info vmware-vum-server[25537] [Originator@6876 sub=EHP] Found cached JWT
2022-11-22T18:23:12.573-06:00 error vmware-vum-server[25537] [Originator@6876 sub=EHP] Response from 192.168.1.1/api/v1/node/services/install-upgrade: HTTP Status:403 'Forbidden'
2022-11-22T18:23:12.573-06:00 error vmware-vum-server[25537] [Originator@6876 sub=EHP] Failed to call NSX-T/api/v1/node/services/install-upgrade
2022-11-22T18:23:12.573-06:00 error vmware-vum-server[25537] [Originator@6876 sub=EHP] Caught exception while finding Nsxt Upgrade Coordinator: Failed to call NSX-T/api/v1/node/services/install-upgrade

• API call GET /api/v1/trust-management/oidc-uris to NSX-T manager shows the lists of OIDC endpoints:

  "results" : [ {
    "oidc_uri" : "https://<VC-FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration",
    "thumbprint" : "<UUID>",
    "oidc_type" : "vcenter",
    "issuer" : "https://<VC-FQDN>/openidconnect/vsphere.local",
    "jwks_uri" : "https://<VC-FQDN>/openidconnect/jwks/vsphere.local",
    "token_endpoint" : "https://<VC-FQDN>/openidconnect/token/vsphere.local",
    "claims_supported" : [ ],
    "override_roles" : [ ],
    "resource_type" : "OidcEndPoint",

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX-T

Cause

This issue occurs when the domain name registered as the OIDC endpoint (Compute manager) case sensitivity is different from the token issuer.
From the log entry we see vSphere.local for the vpxd-extension and Everyone group.
OIDC issuer is registered with all lowercase entry vsphere.local, as can be seen in the issue line of the API response.

Resolution

This is a known issue impacting NSX-T data center.

Workaround:
The SSO domain for vCenter will need to be repointed to correct to conflicting case sensitivity issue.
If you require assistance with this issue, please open a support request with VMware Infrastructure Service Management team for vCenter/PSC.