Application Pod stuck in Pending ErrImagePull state due to Certificate Signed by Unknown Authority: ErrImagePull

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• A cluster's application pod is failing with ErrImagePull and an error message stating that it cannot pull the image due to an untrusted certificate authority.
• An application pod can be deployed on a workload cluster to run on a workload cluster node/VM.
• Workloads in vSphere Pods can be deployed on a Supervisor cluster to run on an ESXi host.
• This issue can occur regardless of whether or not the cluster that the application pod is running on is managed by Tanzu Mission Control (TMC).

While connected to the cluster context where the application pod is running, the following symptoms are observed:

• The failing application pod is in Pending or ErrImagePull state:
  

  kubectl get pods -n <application namespace>

• When describing the failing application pod, an error message similar to the following is present:
  

  kubectl describe pod -n <application namespace> <application pod name>
  
  Status: Pending
  Reason: ErrImagePull
  Message: failed to pull images: failed to get images: Image <application image> has failed. Error: Failed to resolve on node <node name>. Reason: Http request failed. Code 400: ErrorType(3) failed to do request: Head "<private registry>": tls: failed to verify certificate: x509: certificate signed by unknown authority: ErrImagePull

Environment

VMware vSphere Kubernetes Service

Cause

If the application pod is within a workload cluster, the workload cluster is not configured to trust the registry's certificate.

If the application pod is a vSphere Pod deployed in the Supervisor cluster, the Supervisor cluster or ESXi hosts do not trust the registry's certificate.

Resolution

The YAML for the workload cluster will need to have the CA added to its additionalTrustedCAs section.
Note: Change to the additionalTrustedCAs may result in a rolling redeployment of all nodes in the edited workload cluster.

See the below documentation according to the clusterClass:

ClusterClass builtin-generic

• ClusterClass OsConfiguration Documentation
• ClusterClass v3.4.0 YAML Example

ClusterClass tanzukubernetescluster / v1beta1

• v1beta1/v1beta2 Example: Cluster with Additional Trusted CA Certificates for SSL/TLS
• Integrate VKS Clusters with a Private Container Registry

For deploying workloads in vSphere Pods in the Supervisor cluster and ESXi hosts, see the below documentation:

Deploying Workloads to vSphere Pods

Additional Information

Create a Pod from a Container Image in a Private Registry