"API Error: Failed to register management cluster with configurations: Internal Server Error: please try again later (internal error)" in Tanzu Mission Control Self-Managed

Products

• VMware Tanzu Mission Control Self-Managed

Issue/Introduction

• In this scenario, you are unable to register your Supervisor or Management Cluster and you cannot proceed after Step (2) in Tanzu Mission Control Self-Managed 1.4 in an Air-Gapped environment. 

• You will observe that cluster-agent-service pod logs in tmc-local namespace shows there is a problem related to tls certification validation, CA certificates configuration for harbor registry should be an issue if you are using an air-gapped installation.

"error","msg":"failed to get spec for extension-updater","peer.address":"192.X.X.X:34670","request-id":"11c9eef5-5bd4-xxxxxxxxxxx8","request.kind":"unary","span.kind":"server","subcomponent":"requests","system":"grpc","time":"2025-01-02T11:42:56Z","uid":"2e0077b1-cb67-48ba-xxxxxxxxxx"}
{"component":"server-serve-grpc","error":"harbor-registry-xyz.com/harbor-project/498xxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/extensions/vsphere-resource-retriever/manifest:20240730024756960-xxxxxxxxxx was not fetched: could not load image source: get image from the registry: Get \"https://harbor-registry-xyz.com/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown 

• Also, auth-manager-server pods running in tmc-local namespace shows the below error : 

level=error msg="Could not generate tokens using refresh token" X-Request-ID=05e99458-xxxxxxxxx error="oauth2: cannot fetch token: 500 Internal Server Error\nResponse: " http.host=auth.tmc-dns-zone.xxxxxxx http.proto_major=2 http.request.length_bytes=114 http.request.method=POST http.request.referer= http.request.user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" http.url.path=/api/v1/oauth2/token/refresh peer.address=192.X.X.X peer.port=60088 span.kind=server system=http

Environment

Tanzu Mission Control Self-Managed version 1.4 

Cause

Incorrect Harbor CA certificate has been added under trustedCAs section in values.yaml file that contains the key-values for TMC Self-Managed configuration. 

or

Incorrect or mispelt FQDN used for the harbor-instance used to generate the yaml.

 

Resolution

• Add the correct Harbor CA certificate  in values.yaml .

trustedCAs:
   harbor-ca.pem: | # root CA cert of Harbor, if not a well-known CA and if different from the local-ca.pem
     -----BEGIN CERTIFICATE-----
     ...
     -----END CERTIFICATE-----

• Update the package again.

 tanzu package installed update tanzu-mission-control -p tmc.tanzu.vmware.com --version "{{version}}" --values-file {{values.yaml}} --namespace tmc-local

 

• Register the supervisor cluster . 

Additional Information

To get more detailed steps , please check TMC Self Managed installation docs.

https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-mission-control/1-4/tanzu-mission-control-documentation/tanzumc-sm-install-config-install-tmc-sm.html