Containerd TLS Verification Failure and Image Pull Errors Following Certificate Rotation in VKS

Products

• VMware vSphere Kubernetes Service
• VMware Cloud Foundation

Issue/Introduction

In VKS 3.6.0 and later versions, certificates configured through the Cluster variable additionalTrustedCAs are propagated in-place to the machines. If a certificate is rotated and the variable is updated, pods may reach a state of failure accompanied by image pull errors.

Inspecting the events or logs of the affected pods will show the following containerd error: trust: tls: failed to verify certificate: x509: certificate signed by unknown authority

Environment

• vSphere Kubernetes Service 3.6.0
  
  
• vSphere Kubernetes Service 3.7.0

Cause

The in-place mechanism used to update certificates does not restart the containerd service after the system certificates have been updated on the machine. Consequently, containerd retains the previous certificate context in memory and rejects connections utilizing the newly rotated certificates.

Resolution

The reported issue has been addressed and resolved in VKS release 3.7.1 and later versions.

Workaround:

To resolve this issue, the containerd service must be manually restarted on the machines experiencing the failure to force a reload of the system trust store.

1. Establish an SSH session to the affected machine experiencing the image pull failures.

2. Elevate it to root access.

3. Run the following command to restart the containerd service

systemctl restart containerd

4. Check the status of the service to ensure it is running correctly

systemctl status containerd

5. Monitor the failing pods to confirm that image pulls are now completing successfully.