token-principal-identities failed with response code 400 error experienced when attempting to deploy supervisor cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting to deploy supervisor cluster, the following error is seen:

Intitialized vSphere Resources
Configuration error (since XX/XX/XXXX, 00:00:00 AM/PM)
HTTP request error occurred PST http://localhost:1080/external-tp/http1/###.###.###.###/443/.../api/v1/trust-management/token-principal-identities failed with response code 400.

Additionally, logs similar to the following can be found in the wcpsvc.log:

####-##-##T00:00:00.00Z error wcp [kubelifecycle/controller.go:623] [opID=xxxxxxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] Failed to create WCP service PI in NSX. Err: failed to create WCP Service Principal Identity: NSX Principal Identity creation failed: error sending HTTP request: Post "http://localhost:1080/external-tp/http1/###.###.###.###/443/.../api/v1/trust-management/token-principal-identities": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
POST http request failed. URL: http://localhost:1080/external-tp/http1/###.###.###.###/443/.../api/v1/trust-management/token-principal-identities. Status Code: 400. Status: 400 Bad Request
error sending HTTP request: Post "http://localhost:1080/external-tp/http1/###.###.###.###/443/.../api/v1/trust-management/token-principal-identities": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
POST http request failed. URL: http://localhost:1080/external-tp/http1/###.###.###.###/443/.../api/v1/trust-management/token-principal-identities. Status Code: 503. Status: 503 Service Unavailable

The following is seen in update-controllery/sync.logs:

####-##-##T00:00:00.00Z DEBUG sso_domain_change: No updates required in file: /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist  
####-##-##T00:00:00.00Z DEBUG sso_domain_change: Updating SSO domain in file: /usr/lib/vmware-wcp/objects/namespace-operator/90-namespace-operator/namespace-operator.yaml
...
####-##-##T00:00:00.00Z DEBUG sync: Sync done: {"value": [{"type": "GuestCustomized", "status": "TRUE", "reason": "", "messages": [], "severity": "", "lastTransitionTime": "####-##-##T00:00:00.00Z"}, {"type": "ManagementNetworkConfigured", "status": "TRUE", "reason": "", "messages": [], "severity": "", "lastTransitionTime": "####-##-##T00:00:00.00Z"}, {"type": "ConfiguredAsK8sNode", "status": "TRUE", "reason": "", "messages": [], "severity": "", "lastTransitionTime": "####-##-##T00:00:00.00Z"}, {"type": "WorkloadNetworkConfigured", "status": "TRUE", "reason": "", "messages": [], "severity": "", "lastTransitionTime": "####-##-##T00:00:00.00Z"}], "config_modify_time_ns": ###################}. Retry required True 
Although it shows no updated required in the file but the retry required is set to True 

Resource overutilization by NSX managers may also be experienced.

Environment

vCenter 8.0
NSX 4.1.2.0

Cause

This is a known issue related to the sso_domain change module within the supervisor control plane node, which causes repeated attempts to connect to NSX and creation of principle identities and fails.

Resolution

Issue is resolved in 8.0U3e