After Updating Private Registry Certificate, Guest Cluster Image Pulls Fail with “x509: certificate signed by unknown authority”

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Certificates for a Private Container Registry have been recently updated and post that the pods running on the Guest Cluster are unable to pull the image and following error is observed - 
  

  crictl pull <Image Name> 
  
  HH:MM:SS 1064617 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"<FQDN_of_Container_Registry>/<container name>:2024.1_interp\": failed to do request: Head \"https://<URL of Container Registry from where Image is being pulled>\": tls: failed to verify certificate: x509: certificate signed by unknown authority" image="<Image Name>"
  FATA[0000] pulling image: failed to pull and unpack image "<Image Name>": failed to do request: Head "https://<URL of Container Registry from where Image is being pulled>": tls: failed to verify certificate: x509: certificate signed by unknown authority

   
• Guest Cluster nodes (both control plane and worker) are unable to pull container images.

Environment

VMware vSphere Kubernetes Service

Cause

The updated certificate of the external container registry was not injected into the Guest Cluster. As a result, the container runtime (containerd) on Guest Cluster nodes does not trust the new certificate, leading to TLS verification failure while pulling images.

This typically occurs in TKGS clusters that use ClusterClass based configuration, where the new certificate must be explicitly added to the cluster spec.

Resolution

• To resolve the issue, inject the new certificate into the Guest Cluster so that all nodes trust the new container registry certificate.
  Follow the steps as per the Cluster API type from the following documentation :  
  
  
  • For v1alpha3 API - Integrate TKG Service Clusters with a Private Container Registry for v1alpha3 clusters.
  • For v1beta1 API - Integrate TKG Service Clusters with a Private Container Registry for v1beta1 clusters.
    
    
• Perform a Rolling Update of the Guest Cluster so that the new certificate gets propagated to all Guest Cluster Nodes. Refer to the following documentation on how to perform a user initiated Rolling Update - User Initiated Rolling Update.