Tanzu Hub failed to create the organization group due to an internal server error

Products

• VMware Tanzu Platform Core
• VMware Tanzu Platform - Hub

Issue/Introduction

Attempting to create an org group in Tanzu Hub fails with error:

Failed to create the organization '<Org Group name>' due to an internal server error on the foundation '<Foundation name>'. Please verify that the foundation is healthy and try again. Contact your administrator if the problem persists.

All processes in EAR/CF and Platform Service/Hub TAS Collector deployments are running and healthy.

Cause

The ensemble_stitching process on Platform Services deployment shows the same "internal service" error while performing the Token exchange

error: trace='-',parent='-',span='-',user='#####,org='-',source=-,origin=- Token exchange error: Failed to perform token exchange: Response code 500 (Internal Server Error) (UAA endpoint: https://<UAA Endpoint>)

See Platform Service token exchange for further details

The EAR UAA logs explain the reason for the internal server error. The EAR UAA cannot connect to Tanzu Hub UAA 

uaa - 12 [https-jsse-nio-8443-exec-8] - [################ce8e0f,########ce8e0f] .... DEBUG --- FilterChainProxy: Secured POST /error500
uaa - 12 [https-jsse-nio-8443-exec-8] - [################ce8e0f,########ce8e0f] .... ERROR --- HomeController: Internal error
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<Hub URL>/auth/z/default/.well-known/openid-configuration": <Hub URL>: Name or service not known
        at org.springframework.web.client.RestTemplate.createResourceAccessException(RestTemplate.java:926) ~[spring-web-6.2.16.jar!/:6.2.16]

.....
.....
Caused by: java.net.UnknownHostException: <Hub URL>: Name or service not known

Resolution

Check the connectivity from EAR UAA VM to Tanzu Hub UAA

bosh -d <CF Deployment> ssh uaa/0
curl -v <Hub URL>

If it is an internet restricted environment with connectivity only through Http Proxy, ensure EAR Tile---> UAA has a HTTP proxy configured