vSphere Kubernetes Cluster Image Pull Failures from Embedded Harbor Registry after/post-upgrade of Supervisor cluster

Products

• VMware vSphere 7.0 with Tanzu
• VMware vSphere Kubernetes Service
• vSphere with Tanzu
• Tanzu Kubernetes Runtime

Issue/Introduction

After a Supervisor cluster upgrade completed post upgrade of vCenter Server, applications are running into image errors when pulling from the Embedded Harbor registry configured in the environment.

 

While connected to the affected vSphere Kubernetes cluster context, the following symptoms are observed:

• There is a new image-pull secret for the Embedded Harbor Registry that was created shortly after the upgrade completed:

  • kubectl get secrets -n <cluster namespace>

 

Environment

vSphere with Tanzu 7.0 after upgrade from vCenter Server 7.03L (build number 21477706)

vSphere with Tanzu 8.0 after upgrade from vCenter Server 8.0uC (build number 21457384)

Cause

The image-pull-secret for the Embedded Harbor Registry was automatically recreated during the Supervisor cluster upgrade.

This secret was recreated due to an upgrade of the vRegistry component managing the Embedded Harbor Registry service in the environment.

A change was made to resolve security concerns which requires a refresh of the secrets and the kubeconfig configured for the cluster and Embedded Harbor Registry.

Resolution

The newly created secret for the Embedded Harbor Registry will need to be used in a new kubeconfig.

Follow the steps in the below TechDocs Documentation to:

1. Create a new kubeconfig from the newly generated image-pull-secret
   
   
2. Create a new registry service secret based on the new saved image-pull secret
   
   
3. Re-configure the pods pulling from the Embedded Harbor Registry to use the new imagePullSecret

TechDocs Documentation: Use the Embedded Harbor Registry with Tanzu Kubernetes Clusters