Supervisor cluster deletion fails with error "HTTP request error occurred. GET https://nsx:443/api/v1/trust-management/principal-identities failed with response code 500"

Products

• VMware vSphere Kubernetes Service
• VCF Operations

Issue/Introduction

• When attempting to deactivate or delete a vSphere with Tanzu Supervisor Cluster within a VMware Cloud Foundation environment, the task fails or remains in a permanent pending state.
• Review of the /var/log/vmware/wcp/wcpsvc.log file on the vCenter Server shows log traces similar to the following:

YYYY-MM-DDT HH:MM:SS debug wcp [identity/principal_identity.go:333] [opID=] NSX HTTP Response is: &{500 Internal Server Error 500 HTTP/1.1 1 1 map[...] ...}, {"module_name":"common-services","error_message":"Internal server error has occurred.","details":"I/O error on GET request for \"https://<vCenter Server FQDN>/openidconnect/jwks/vsphere.local\": <Thumbprint Hash>; nested exception is javax.net.ssl.SSLHandshakeException: <Thumbprint Hash>","error_code":99}
YYYY-MM-DDT HH:MM:SS error wcp [identity/principal_identity.go:240] [opID=] Failed to GET cert based PI from NSX managers. Err: GET http request failed. URL: https://<NSX Manager FQDN>:443/api/v1/trust-management/principal-identities. Status Code: 500. Status: 500 Internal Server Error
YYYY-MM-DDT HH:MM:SS warning wcp [kubelifecycle/controller.go:417] [opID=] Unable to disable cluster domain-c8. Err failed cleaning NSX-T resources due to failure to fetch supervisor <Cluster UUID> principal identity: NSX cluster 'wcp-cluster-user-<Cluster UUID>' principal identity fetch failed: GET http request failed. URL: https://<NSX Manager FQDN>:443/api/v1/trust-management/principal-identities. Status Code: 500. Status: 500 Internal Server Error

• When reviewing the NSX Manager User Interface under System > Fabric > Compute Managers, the connection status for the associated vCenter Server shows as Down.

Environment

• VMware Cloud Foundation
• VMware vSphere with Tanzu
• VMware NSX

Cause

• This issue occurs when the SSL certificate on the vCenter Server has been updated, replaced, or modified, but the new certificate thumbprint has not been synchronized inside the NSX Manager Compute Managers registry.
• During the Supervisor Cluster deactivation workflow, the vCenter Workload Control Plane (WCP) service queries the NSX Principal Identities API. NSX Manager then attempts to authenticate this call by performing an internal validation check against the vCenter Server OIDC JWKS endpoint (/openidconnect/jwks/vsphere.local). Because the certificate thumbprint trusted by NSX does not match the active certificate presented by vCenter Server, a javax.net.ssl.SSLHandshakeException occurs internally within NSX, causing the endpoint to return an HTTP 500 error back to the WCP service.

Resolution

To resolve this issue, the updated vCenter Server certificate thumbprint must be accepted and saved inside NSX Manager:

1. Open a web browser and log into the NSX Manager User Interface.

2. Navigate to System > Fabric > Compute Managers.

3. Identify the affected vCenter Server from the inventory display.

4. Select the entry and click Edit.

5. Within the configuration window, look for the alert banner detailing a certificate/thumbprint mismatch detection.

6. Click the Update link provided within the alert text to download and trust the current vCenter Server certificate.

7. Click Save to confirm and apply the modification.

8. Verify that the connection status for the vCenter Compute Manager updates to Up.

9. Return to the vCenter Server interface and re-initiate or resume the Supervisor Cluster deactivation workflow.