The pod, smart-warmer-image-fetcher, hits an error "failed to register layer: max depth exceeded"

Products

• VMware Tanzu Build Service

Issue/Introduction

The pod, smart-warmer-image-fetcher, in namespace build-service could failed with the following error when deploying with AWS EKS.

Warning Failed 6m58s (x555 over 4d22h) kubelet Failed to pull image "harbor.example.net/tbs/tbs:clusterbuilder-full@sha256:######": rpc error: code = Unknown desc = failed to register layer: max depth exceeded Warning Failed 6m58s (x555 over 4d22h) kubelet Error: ErrImagePull 

Environment

Product Version: Other

Resolution

The similar issue is not observed in other platform for example vSphere TKG. We are reporting this issue to the engineering team and hopefully this can be fixed soon.

As a walkaround, we could switch the containerRuntime from docker to containerd. This can be done by create a new NodeGroup with "--container-runtime containerd" included.

Here is an example of a configuration file.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: $CLUSTER_NAME
  region: $REGION
managedNodeGroups:
  - name: $NODEGROUP_NAME
    ami: $AMI_ID
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh $CLUSTER_NAME --container-runtime containerd
EOF

Once new worker nodes are created, we can drain and remove the old ones gradually in order to migrate the pods to adopt the new container runtime.