Tanzu Mission Control Pods are in ImagePullBackOff with "Access Denied" error
search cancel

Tanzu Mission Control Pods are in ImagePullBackOff with "Access Denied" error

book

Article ID: 426333

calendar_today

Updated On:

Products

VMware Tanzu Mission Control - SM

Issue/Introduction

  • Workload Clusters are in Disconnected state in Tanzu Mission Control Self Manage (TMC-SM).

  • The TMC-SM is in unhealthy state.
  • One or more TMC pods in the tmc-local namespace on the workload cluster where TMC-SM is installed is in "ImagePullBackOff" and the pods was restarted recently.

    #kubectl get pods -n tmc-sm-local

    inspection-server-7558d6fdd-dnn4l                    2/2     Running            1 (62d ago)     62d
    inspection-server-7558d6fdd-lcgg6                    2/2     Running            0               62d
    intent-server-6b5cbb8ddf-2crrr                       0/1     ImagePullBackOff   1               2d
    intent-server-6b5cbb8ddf-vpbqs                       1/1     Running            0               62d
    intent-server-6b5cbb8ddf-w2g5d                       1/1     Running            1 (62d ago)     62d
    kafka-controller-0                                   1/1     Running            0               62d

  • One or more TMC pods in the vmware-system-tmc  namespace on the workload cluster that in disconnected state is in "ImagePullBackOff", and the pods was restarted recently.

    # kubectl get po -n vmware-system-tmc
    NAME                                   READY   STATUS             RESTARTS       AGE
    agent-updater-56495b7dc5-bpk2h         1/1     Running                0          62d
    agentupdater-workload-29434672-xhfcg   0/1     ImagePullBackOff       1          2d
    extension-manager-c58857d7f-f6k7q      0/1     ImagePullBackOff       1          6m50
    sextension-updater-58578dcdcc-9wdz2    1/1     Running                0          62d

  • The harbor is in running state.
  • The image exist in the "tmc-sm" project on harbor.
  • The describe of the tmc pod that inImagePullBackOff state showing that pull of the image from the Harbor is faling with "authorization failed: no basic auth credentials" error.

    #kubectl describe pod -n <pod-name> -n <tmc-namespace>
                             
    Events:
      Type     Reason     Age                From               Message
      ----     ------     ----               ----               -------
      Normal   Scheduled  77s                default-scheduler  Successfully assigned tmc-local/agentupdater-workload-29434672-xhfcg to workernode-md-0-r9m87-6kmqz-ft4rb
      Normal   Pulling    38s (x3 over 77s)  kubelet            Pulling image "harbor-url/tmc-sm/package-repository@sha256:e67e5dd6198307199da790c7a5bac860a66603892c97ac08ba61805b7a28fceb"
      Warning  Failed     38s (x3 over 77s)  kubelet            Failed to pull image "harbor-url/tmc-sm/package-repository@sha256:e67e5dd6198307199da790c7a5bac860a66603892c97ac08ba61805b7a28fceb": failed to pull and unpack image "harbor-url/tmc-sm/package-repository@sha256:e67e5dd6198307199da790c7a5bac860a66603892c97ac08ba61805b7a28fceb": failed to resolve reference "harbor-url/tmc-sm/package-repository@sha256:e67e5dd6198307199da790c7a5bac860a66603892c97ac08ba61805b7a28fceb": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials
      Warning  Failed     38s (x3 over 77s)  kubelet            Error: ErrImagePull
      Normal   BackOff    1s (x5 over 77s)   kubelet            Back-off pulling image "harbor-url/tmc-sm/package-repository@sha256:e67e5dd6198307199da790c7a5bac860a66603892c97ac08ba61805b7a28fceb"
      Warning  Failed     1s (x5 over 77s)   kubelet            Error: ImagePullBackOff

  • When ssh to the node where the the TMC pod is located and trying to pull the image we get the same pervious error

Environment

Tanzu Mission Control Self Manage (TMC-SM)

Cause

  • The image are faling to get pulled since the "tmc-sm" project on harbor which hold the TMC images is set to "Privet" which required authenticated registries and it is currently not supported in TMC-SM.
  • Tanzu Mission Control Self Manage (TMC-SM)  currently only support a public project is required in Harbor. see "Preparing your cluster to host Tanzu Mission Control Self-Managed". 

  • We are able to pull the image fine using the admin user name and password

    `crictl pull -u admin  harbor-url/tmc-sm/package-repository@sha256:e67e5dd6198307199da790c7a5bac860a66603892c97ac08ba61805b7a28fceb            DEBU[0000] get image connection

    Enter Password:

    DEBU[0008] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:harbor-url/tmc-sm/package-repository@sha256:e67e5dd6198307199da790c7a5bac860a66603892c97ac08ba61805b7a28fceb,Annotations:map[string]string{},},Auth:&AuthConfig{Username:admin,Password:<pass>,Auth:,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,}
    DEBU[0010] PullImageResponse: &PullImageResponse{ImageRef:sha256:01e9f8fdf51f87678ce7592dd8c76e3551f7e354937dca6e3d5e4b804e89c957,}
    Image is up to date for sha256:01e9f8fdf51f87678ce7592dd8c76e3551f7e354937dca6e3d5e4b804e89c957

Resolution

  • Change the tmc-sm project from "Private" to"Public" on the harbor .

    1. Log in to the Harbor interface with an account that has at least project administrator privileges.
    2. Go to Projects and select a project.
    3. Select the Configuration tab.
    4. To make all repositories under the project accessible to everyone, select the Public checkbox.

  • Restart the tmc pods that in ImagePullBackOff state on the TMC cluster where the TMC -SM is installed  to get them back into running state.
  • Restart  tmc pods that in ImagePullBackOff on the workload clusters that in disconnected state to get them back into running state.


 

Additional Information

  • If the workload clusters still showing disconnected after restarting the TMC pods 
  • The TMC pods are running on the TMC-SM cluster.
    Then:
    • To attach a cluster to your VMware Tanzu Mission Control organization that was previously attached and subsequently fell into a disconnected state.. see  Reattach a Cluster 
    • Tanzu cli can be used to reattche the disconnected cluster See  Reattach an existing cluster 

# tanzu tmc cluster reattach clusgter-name -p <provisioner-name> -m <management-cluster-name>


Notes:

Powered by Wolken Software