https://registry.tkg.vmware.run/v2/pause/manifests/3.2": x509: certificate has expired or is not yet valid

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)

Issue/Introduction

This error message can occur when images previously pulled from the deprecated 'https://registry.tkg.vmware.run/' registry are no longer available on a node. On TKGi, these images are built into the tile and do not need to be pulled externally. Since this repository is no longer in use, there are no plans to renew its certificate.

You may notice that pods are failing to schedule on a specific node and see events similar to the following snippet:

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Warning  FailedCreatePodSandBox  5m58s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "registry.tkg.vmware.run/pause:3.2": failed to pull image "registry.tkg.vmware.run/pause:3.2": failed to pull and unpack image "registry.tkg.vmware.run/pause:3.2": failed to resolve reference "registry.tkg.vmware.run/pause:3.2": failed to do request: Head "https://registry.tkg.vmware.run/v2/pause/manifests/3.2": x509: certificate has expired or is not yet valid: current time 2024-04-04T07:53:46Z is after 2024-03-26T23:59:59Z

Cause

The images are missing due to manual deletion or removal during a node's disk pressure event

Resolution

Follow the below steps to resolve this issue: 

1. SSH to the problematic node
   • bosh -d <service-instance-ID> ssh <problematic-node>
2. Switch to root user
   • sudo -i
3. Execute the following script
   • /var/vcap/jobs/load-images/bin/post-start

After the above steps are followed, the images will be available once again.