Failed to run health checks for NSX-T

Products

VMware NSX VMware vCenter Server

Issue/Introduction

Health Checks ran on vCenter to upgrade the ESXi hosts and the receive the following error:
- Failed to run health checks for NSX-T on '<Hostname>'.
- Failed to run health checks for NSX-T on '<Hostname>'.
- Health Check for '<Hostname>' failed
Running health checks for NSX-T on vLCM cluster fails with error: Failed to run health checks for NSX-T

Log lines similar to the below are encountered on vCenter Server in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log:

error vmware-vum-server[18363] [Originator@6876 sub=EHP] Response from localhost/external-tp/http1/###.###.###.###/443/########################################/api/v1/node/services/install-upgrade: HTTP Status:403 'Forbidden'
error vmware-vum-server[18363] [Originator@6876 sub=EHP] Failed to call NSX-T /external-tp/http1/###.###.###.###/443/########################################/api/v1/node/services/install-upgrade
error vmware-vum-server[18363] [Originator@6876 sub=EHP] Caught exception while finding Nsxt Upgrade Coordinator: Failed to call NSX-T /external-tp/http1/<NSX-Manager-IP>/443/775EFE6BEB018CF6A7C2FAA47B3870F8B2D74EC3/api/v1/node/services/install-upgrade

Log lines similar to the below are encountered on NSX Manager in /var/log/proxy/reverse-proxy.log:

INFO grpc-default-executor-219 AuthService 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] check
INFO grpc-default-executor-219 HttpClientUtil 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Making request to http://###.###.###.###:6565/api/v1/node/services/install-upgrade
ERROR Processing request ########-####-####-####-############ VcAuthTokenInterceptingFilter 5890 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] Unable to find the issuer https://<vCenter Server FQDN>/openidconnect/vSphere.LOCAL in the OIDC Endpoint TrustStore
INFO Processing request ########-####-####-####-############ OidcEndPointTrustStoreInMemory 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] ===== OidcEndPointTrustStore contents =====
INFO Processing request ########-####-####-####-############ OidcEndPointTrustStoreInMemory 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Issuer https://<vCenter Server FQDN>/openidconnect/vsphere.local, uri https://<vCenter Server FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration and thumbprint ################################################################
INFO Processing request ########-####-####-####-############ BaseProxyDelegate 5890 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Processing request to /api/v1/node/services/install-upgrade with /api/v1/node -> ###.###.###.###:7441:/api/v1/node

In NSX UI under "System" "Fabric" "Compute Manager" the vCenter connection may also show down.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

VMware NSX

Cause

vCenter generated JWT tokens, which have a different issuer than what NSX has discovered as the issuer for vCenter.

Example:

NSX has discovered: https://<vCenter Server FQDN>/openidconnect/vsphere.local
But the vCenter generated tokens have the issuer as: https://<vCenter Server FQDN>/openidconnect/vSphere.LOCAL

Resolution

Workaround

In NSX UI, under "System", "Fabric", "Computer Managers", edit the compute manager (vCenter) connection by first click on "edit" next to "FQDN or IP address" then entering the desired vCenter Server username and password.
- Ensure "Enable Trust" is enabled.
- "Create Service Account" is optional.
You may be prompted for a different vCenter thumbprint, accept and save it.

If above steps did not resolve the issue, continue with the following:

Get the OIDC config in NSX via API call: GET https://<NSX-Manager-iP>/api/v1/trust-management/oidc-uris
- save this API output
Example:
{
    "results": [
        {
            "oidc_uri": "https://<vCenter Server FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration",
            "thumbprint": "<SSL Thumbprint>",
            "oidc_type": "vcenter",
            "scim_endpoints": [],
            "claim_map": [],
            "serviced_domains": [],
            "restrict_scim_search": false,
            "end_session_endpoint_uri": "https://<vCenter Server FQDN>/openidconnect/logout/vsphere.local",
            "issuer": "https://<vCenter Server FQDN>/openidconnect/vsphere.local",
            "jwks_uri": "https://<vCenter Server FQDN>/openidconnect/jwks/vsphere.local",
            "token_endpoint": "https://<vCenter Server FQDN>/openidconnect/token/vsphere.local",
            "claims_supported": [],
            "override_roles": [],
            "resource_type": "OidcEndPoint",
            "id": "<OIDC-URI-ID>",
            "display_name": "<OIDC-URI-ID>",
            "_create_time": 1675162497329,
            "_create_user": "<user>",
            "_last_modified_time": 1675162497329,
            "_last_modified_user": "<user>",
            "_system_owned": false,
            "_protection": "NOT_PROTECTED",
            "_revision": 0
        }
    ]
}
Delete the OIDC URI via API call: DELETE /api/v1/trust-management/oidc-uris/<OIDC-URI-ID>
Modify the API output saved from step 1 to remove the extra fields and update "oidc_uri" to use vSphere.LOCAL instead of vspehre.local.

Example:

{
            "oidc_uri": "https://<vCenter Server FQDN>/openidconnect/vSphere.LOCAL/.well-known/openid-configuration",
            "thumbprint": "<thumbprint>",
            "oidc_type": "vcenter",
            "scim_endpoints": [],
            "claim_map": [],
            "serviced_domains": [],
            "restrict_scim_search": false,
            "end_session_endpoint_uri": "https://<vCenter Server FQDN>/openidconnect/logout/vsphere.local",
            "issuer": "https://<vCenter Server FQDN>/openidconnect/vsphere.local",
            "jwks_uri": "https://<vCenter Server FQDN>/openidconnect/jwks/vsphere.local",
            "token_endpoint": "https://<vCenter Server FQDN>/openidconnect/token/vsphere.local",
            "claims_supported": [],
            "override_roles": [],
            "resource_type": "OidcEndPoint",
            "id": "<OIDC-URI-ID>",
            "display_name": "<OIDC-URI-ID>"
}

Your output may look different, the fields can be removed are the ones start with underscore "_".

Create a new OIDC via POST API call with the modified output from step 5 as body:
- POST https://<NSX-manager-IP>/api/v1/trust-management/oidc-uris/

Note: If the compute manager is updated, it will override the OIDC changes and the same workaround has to be applied after the compute manager is updated.

If you believe you have encountered this issue and are unable to implement a workaround, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.

Provide:

vCenter Server support bundle.
NSX Manager support bundle.

Handling Log Bundles for offline review with Broadcom support.