Error: "tls: failed to verify certificate: x509..." leading to ImagePullBackOff for application pods

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Several application pods fail to start and remain stuck in an ImagePullBackOff or Init:ImagePullBackOff state.
• Describing the pod : k describe pod <pod_name> -n <namespace_name> | less

Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  Failed   32m (x22 over 110m)     kubelet  (combined from similar events): Failed to pull image "<image-registry-fqdn>/registry/image:SHA-digest": failed to pull and unpack image "<registry-fqdn>/registry/image:SHA-digest": failed to resolve reference "<image-registry-fqdn>/registry/image:SHA-digest": failed to do request: Head "https:<image-registry-fqdn>/registry/image:SHA-digest": dial tcp: lookup <image-registry-fqdn> on 127.0.0.53:53: read udp 127.0.0.1:39483->127.0.0.53:53: i/o timeout
  Normal   BackOff  3m21s (x498 over 127m)  kubelet  Back-off pulling image "image-registry-fqdn>/registry/image:SHA-digest"
  Warning  Failed   2m39s (x499 over 127m)  kubelet  Error: ImagePullBackOff

• crictl pull <image-registry-fqdn>/registry/image:SHA-digest

"PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"<image-registry-fqdn>/registry/image:SHA-digest\": failed to resolve reference \"<image-registry-fqdn>/registry/image:SHA-digest\": failed to do request: Head \"https://<image-registry-fqdn>/registry/image:SHA-digest\": tls: failed to verify certificate: x509: certificate is valid for <image-registry-1>, not <image-registry-2>" image="<image-registry-fqdn>/registry/image:SHA-digest"

Environment

• VMware vCenter Server 9.x
• VMware vSphere Kubernetes Service

Cause

• This issue is caused by a mismatch between the requested FQDN and the Common Name (CN) or Subject Alternative Name (SAN) presented in the Harbor SSL certificate. 
• The Kubernetes `kubelet` and `containerd` runtime require the correct hostname in the URL to match the identity proven by the certificate to complete a secure TLS handshake.

Resolution

• Troubleshoot the connectivity between the VKS Guest Cluster Control Plane nodes and the external image registry.

Note: External Harbor is out of support scope for GSS.