Supervisor cluster reconciliation fails due to time-out waiting for capi-controller-manager

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

The kube-rbac-proxy container fails to get created for capi-controller-manager

Error in vSphere UI > Workload management page:

Finished unsuccessfully (Reconcile failed: (message: kapp: Error: Timed out waiting after 15m0s for resources: [deployment/capi-controller-manager (apps/v1) namespace: svc-tkg-domain-####])).

We get the below messages when describing the capi-controller-manager pod:

kubectl describe pod capi-controller-manager-####-####
  
containerStatuses:
  - image: projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.1.0/tkg-service@sha256:dce2722e591b839742041718ca99069efd06717d6f0301fa3061a60965822e04
    imageID: ""
    lastState: {}
    name: kube-rbac-proxy
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'failed to create containerd container: error unpacking image: unexpected
          media type application/octet-stream for sha256:dce2722e591b839742041718ca99069efd06717d6f0301fa3061a60965822e04: not found'
        reason: CreateContainerError
  - containerID: containerd://#################
  
  Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------

  Normal   Started    14m                   kubelet            Started container manager
  Warning  Failed     13m (x11 over 14m)    kubelet            Error: failed to create containerd container: error unpacking image: unexpected media type application/octet-stream for sha256:dce2722e591b839742041718ca99069efd06717d6f0301fa3061a60965822e04: not found
  Normal   Pulled     4m45s (x50 over 14m)  kubelet            Container image "projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.1.0/tkg-service@sha256:dce2722e591b839742041718ca99069efd06717d6f0301fa3061a60965822e04" already present on machine

Environment

• vCenter Server 8.0
• Supervisor version: v1.29.7

Cause

The capi-controller-manager is unable to unpack the image for kube-rbac-proxy container

Resolution

Manually pull the image for the capi-controller-manager rbac-proxy container using the below command:

crictl pull projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.1.0/tkg-service@sha256:dce2722e591b839742041718ca99069efd06717d6f0301fa3061a60965822e04