Tanzu Mission Control Self-Mananged - GKE Cluster attach failure due to local registry configuration issues
Article ID: 381478
Updated On:
Products
Issue/Introduction
Tanzu Mission Control dashboard does not display cluster details or that is available.
You are seeing errors like below in relation to your local registry:
failed to authorize: failed to fetch oauth token: unexpected status from GET request to <registry-url/registry-namespace>
Warning Failed 39s (x3 over 78s) kubelet Failed to pull image
Normal BackOff 11s (x4 over 78s) kubelet Back-off pulling image
Warning Failed 11s (x4 over 78s) kubelet Error: ImagePullBackOff
the extension update logs show errors similar to below:
"extension-handler","error":"registry not found in the docker config","level":"error","msg":"failed to check if the referenced images exist"
The TMC UI may also display below error at the top in red:
API Error: Failed to create local registry credential: image registry credential annotation registry-namespace value is invalid (should not contain any spaces or invalid characters) (invalid argument)
Environment
Tanzu Mission Control
Tanzu Mission Control Self-Managed
Cause
In this case there were a number of causes
- images were not synced to local registry
- registry-namespace was initially blank and not used.
- registry-namespace has a character limit of 55 characters, in this case it was 73 charaters
- also local registry credentials was configured with incorrect access secret where initially using the private cert key which was incorrect.
Resolution
The required images must be synced to the local image registry prior to attaching a GKE cluster.
Syncing the images to the local registry guide can be found as per Sync Images to Your Local Image Registry doc.
The local image registry configuration can be performed via:
- the TMC UI as per Add a Local Image Registry for Tanzu Mission Control
- the TMC CLI as per Add a Local Image Registry for Tanzu Mission Control Using the CLI
- the TMC API as per Add a Local Image Registry for Tanzu Mission Control Using the API
Note - the registry-namespace must not be left blank and it has a character limit of 55 characters, it is also important to note that registry-namespace is in fact the registry-url 's suffix. So for example the registry-url would be mydomain.com and theregistry-namespace or registry-url 's suffix would be what appears after the registry-url, for example: /myregistrynamespace/ where combining both registry-url and registry-namespace would look like: mydomain.com/myregistrynamespace.
Where using the TMC UI to configure the local image registry's RegistryURL, RegistryNamespace, AccessID and AccessSecret is encoded and stored as dockerconfigjson in the resource spec.
It is important to note that the AccessSecret must be the jsonkey file in it's entirety and not a private key alone.
For testing this connection you could test as per below:
cat jsonkey.json | docker login -u _json_key --password-stdin https://mydomain.com/myregistrynamespace
where RegistryURL is mydomain.com, RegistryNamespace is /myregistrynamespace, AccessID is _json_key and AccessSecret is jsonkey.json
Feedback
