Tanzu Mission Control Self-Mananged - GKE Cluster attach failure due to local registry configuration issues

Products

• VMware Tanzu Mission Control
• VMware Tanzu Mission Control Self-Managed
• VMware Tanzu Mission Control - SM
• Tanzu Mission Control

Issue/Introduction

• Tanzu Mission Control dashboard does not display cluster details or that is available.
• You are seeing errors like below in relation to your local registry:
  
  failed to authorize: failed to fetch oauth token: unexpected status from GET request to <registry-url/registry-namespace>

Warning  Failed     39s (x3 over 78s)  kubelet                                Failed to pull image
Normal   BackOff    11s (x4 over 78s)  kubelet                                Back-off pulling image
Warning  Failed     11s (x4 over 78s)  kubelet                                Error: ImagePullBackOff
  
  
• The extension update logs show errors similar to below:
  
  "extension-handler","error":"registry not found in the docker config","level":"error","msg":"failed to check if the referenced images exist"
  
  
• The TMC UI may also display below error at the top in red:
  
  API Error: Failed to create local registry credential: image registry credential annotation registry-namespace value is invalid (should not contain any spaces or invalid characters) (invalid argument)

Environment

• Tanzu Mission Control
• Tanzu Mission Control Self-Managed

Cause

• In this case there were a number of causes:
  
   - Images were not synced to local registry.
   - Registry-namespace was initially blank and not used.
   - Registry-namespace has a character limit of 55 characters, in this case it was 73 characters.
   - Also local registry credentials was configured with incorrect access secret where initially using the private cert key which was incorrect.

Resolution

• The required images must be synced to the local image registry prior to attaching a GKE cluster.
• Syncing the images to the local registry guide can be found as per Sync Images to Your Local Image Registry doc.
  
  
• The local image registry configuration can be performed via:
   - The TMC UI as per Add a Local Image Registry for Tanzu Mission Control
   - The TMC CLI as per Add a Local Image Registry for Tanzu Mission Control Using the CLI
   - The TMC API as per Add a Local Image Registry for Tanzu Mission Control Using the API
  
  Notes :
  
  
  • The registry-namespace must not be left blank and it has a character limit of 55 characters, it is also important to note that registry-namespace is in fact the registry-url 's suffix. So for example the registry-url would be mydomain.com and theregistry-namespace or registry-url 's suffix would be what appears after the registry-url, for example: /myregistrynamespace/ where combining both registry-url and registry-namespace would look like: mydomain.com/myregistrynamespace.
    
    
  • When using the TMC UI to configure the local image registry's RegistryURL, RegistryNamespace, AccessID and AccessSecret is encoded and stored as dockerconfigjson in the resource spec.
    
    
  • It is important to note that the AccessSecret must be the jsonkey file in it's entirety and not a private key alone.
    For testing this connection you could test as per below:
    
    
  • cat jsonkey.json | docker login -u _json_key --password-stdin https://mydomain.com/myregistrynamespace
    
    
  • where RegistryURL is mydomain.com, RegistryNamespace is /myregistrynamespace, AccessID is _json_key and AccessSecret is jsonkey.json