Failed to run health checks for NSX-T

Products

VMware NSX VMware vCenter Server

Issue/Introduction

Running health checks for NSX-T on vLCM cluster fails with error: Failed to run health checks for NSX-T

Log lines similar to the below are encountered on vCenter Server in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log:

error vmware-vum-server[18363] [Originator@6876 sub=EHP] Response from localhost/external-tp/http1/###.###.###.###/443/########################################/api/v1/node/services/install-upgrade: HTTP Status:403 'Forbidden'
error vmware-vum-server[18363] [Originator@6876 sub=EHP] Failed to call NSX-T /external-tp/http1/###.###.###.###/443/########################################/api/v1/node/services/install-upgrade
error vmware-vum-server[18363] [Originator@6876 sub=EHP] Caught exception while finding Nsxt Upgrade Coordinator: Failed to call NSX-T /external-tp/http1/192.168.200.92/443/775EFE6BEB018CF6A7C2FAA47B3870F8B2D74EC3/api/v1/node/services/install-upgrade

Log lines similar to the below are encountered on NSX Manager in /var/log/proxy/reverse-proxy.log:

INFO grpc-default-executor-219 AuthService 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] check
INFO grpc-default-executor-219 HttpClientUtil 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Making request to http://###.###.###.###:6565/api/v1/node/services/install-upgrade
ERROR Processing request ########-####-####-####-############ VcAuthTokenInterceptingFilter 5890 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] Unable to find the issuer https://<vCenter Server FQDN>/openidconnect/vSphere.LOCAL in the OIDC Endpoint TrustStore
INFO Processing request ########-####-####-####-############ OidcEndPointTrustStoreInMemory 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] ===== OidcEndPointTrustStore contents =====
INFO Processing request ########-####-####-####-############ OidcEndPointTrustStoreInMemory 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Issuer https://<vCenter Server FQDN>/openidconnect/vsphere.local, uri https://<vCenter Server FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration and thumbprint ################################################################
INFO Processing request ########-####-####-####-############ BaseProxyDelegate 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Processing request to /api/v1/node/services/install-upgrade with /api/v1/node -> ###.###.###.###:7441:/api/v1/node

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

VMware NSX

Cause

vCenter generated JWT tokens, which have a different issuer than what NSX has discovered as the issuer for vCenter.

Example:

NSX has discovered: https://<vCenter Server FQDN>/openidconnect/vsphere.local
But the vCenter generated tokens have the issuer as: https://<vCenter Server FQDN>/openidconnect/vSphere.LOCAL

Resolution

Workaround 1

Get the OIDC in NSX via API: GET /api/v1/trust-management/oidc-uris
If the returned output is blank, continue to step 3. If it returns data, compare with Workaround 2.
Example of blank data:
{
"results" : [ ]
}
In NSX UI, under System / Computer Managers, edit the vCenter Server connection by entering the vCenter Server username and password.
Ensure "Enable Trust" is enabled.
"Create Service Account" is optional.
Click SAVE and let NSX reconnect to vCenter Server.
Run the previous API call from step 1 again and the correct data should be populated:
{
"results": [
{
"oidc_uri": "https://<vCenter Server FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration",
"thumbprint": "<thumbprint>",
"oidc_type": "vcenter",
"scim_endpoints": [],
"claim_map": [],
"serviced_domains": [],
"restrict_scim_search": false,
"end_session_endpoint_uri": "https://<vCenter Server FQDN>/openidconnect/logout/vsphere.local",
"issuer": "https://<vCenter Server FQDN>/openidconnect/vsphere.local",
"jwks_uri": "https://<vCenter Server FQDN>/openidconnect/jwks/vsphere.local",
"token_endpoint": "https://<vCenter Server FQDN>/openidconnect/token/vsphere.local",
"claims_supported": [],
"override_roles": [],
"csp_config": {
"customer_org_id": "",
"additional_org_ids": []
},
"resource_type": "OidcEndPoint",
"id": "<OidcEndpoint-UUID>",
"display_name": "<OidcEndpoint-UUID>",
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_create_time": 1726784829169,
"_create_user": "<user>",
"_last_modified_time": 1726784829169,
"_last_modified_user": "<user>",
"_revision": 0
}
]
}

Workaround 2

Get the OIDC in NSX via API call: GET /api/v1/trust-management/oidc-uris"
Example:
{
    "results": [
        {
            "oidc_uri": "https://<vCenter Server FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration",
            "thumbprint": "<SSL Thumbprint>",
            "oidc_type": "vcenter",
            "scim_endpoints": [],
            "claim_map": [],
            "serviced_domains": [],
            "restrict_scim_search": false,
            "end_session_endpoint_uri": "https://<vCenter Server FQDN>/openidconnect/logout/vsphere.local",
            "issuer": "https://<vCenter Server FQDN>/openidconnect/vsphere.local",
            "jwks_uri": "https://<vCenter Server FQDN>/openidconnect/jwks/vsphere.local",
            "token_endpoint": "https://<vCenter Server FQDN>/openidconnect/token/vsphere.local",
            "claims_supported": [],
            "override_roles": [],
            "resource_type": "OidcEndPoint",
            "id": "<SAVE THIS OIDC-URI-ID>",
            "display_name": "<SAVE THIS OIDC-URI-ID>",
            "_create_time": 1675162497329,
            "_create_user": "<user>",
            "_last_modified_time": 1675162497329,
            "_last_modified_user": "<user>",
            "_system_owned": false,
            "_protection": "NOT_PROTECTED",
            "_revision": 0
        }
    ]
}
Note the OIDC URI ID from the above output.
Note the "Unable to find the issuer" value from the NSX-T log seen in the issue description section. (https://<vCenter Server FQDN>/openidconnect/vSphere.LOCAL)
Delete the OIDC URI via API call: DELETE /api/v1/trust-management/oidc-uris/<OIDC-URI-ID>

Create a new OIDC via API call: POST /api/v1/trust-management/oidc-uris/
Body:

{
            "oidc_uri": "https://<vCenter Server FQDN>/openidconnect/vSphere.LOCAL/.well-known/openid-configuration", ============>> collected on Step - 3
            "thumbprint": "<thumbprint>",
            "oidc_type": "vcenter",
            "scim_endpoints": [],
            "claim_map": [],
            "serviced_domains": [],
            "restrict_scim_search": false,
            "end_session_endpoint_uri": "https://<vCenter Server FQDN>/openidconnect/logout/vsphere.local",
            "issuer": "https://<vCenter Server FQDN>/openidconnect/vsphere.local",
            "jwks_uri": "https://<vCenter Server FQDN>/openidconnect/jwks/vsphere.local",
            "token_endpoint": "https://<vCenter Server FQDN>/openidconnect/token/vsphere.local",
            "claims_supported": [],
            "override_roles": [],
            "resource_type": "OidcEndPoint",
            "id": "<OIDC-URI-ID from Step 1>",
            "display_name": "<OIDC-URI-ID from Step 1>"
}

Note: If the compute manager is updated, it will override the OIDC changes and the same workaround has to be applied after the compute manager is updated.

If you believe you have encountered this issue and are unable to implement a workaround, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.

Provide:

vCenter Server support bundle.
NSX Manager support bundle.

Handling Log Bundles for offline review with Broadcom support.