TKG Workload Cluster upgrade stuck in DrainingNode stage due to Antrea CNI failure

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Tanzu Kubernetes Grid (TKG) workload cluster upgrade halts while some nodes are updated and others remain in a legacy version.
• The kubectl describe machine output for impacted nodes shows a state of Deleting with the message: Drain not completed yet ... delay likely due to Pods not terminating.
• Antrea agent pods on the affected worker nodes are in ImagePullBackOff or Init:ImagePullBackOff status.
• Describing the Antrea pod reveals it is attempting to pull from an incorrect local registry: Pulling image "localhost:5000/tkg/packages/core/antrea@sha256:########################".
• Pod termination on the node fails with a FailedKillPod warning: failed to "KillPodSandbox" ... plugin type="antrea" failed (delete): rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/antrea/cni.sock: connect: connection refused".

Environment

VMware Cloud Foundation (VCF)
VMware vSphere Kubernetes Service (VKS)

Cause

The exact root cause is not known, as it can have several different reasons, such as a full root disk when the update was started. However, because the Antrea image is missing, the Antrea CNI cannot initialize. The kubelet is then unable to communicate with the CNI to destroy network sandboxes, preventing pods from terminating and blocking the node drain process required for the upgrade to proceed.

Resolution

Manually export the required Antrea image from a healthy, upgraded node and import it into the failing node to restore CNI functionality.

Step 1: Preparation and Disk Space Verification

1. Identify a working node that has already successfully upgraded to the target version.
2. Identify the failing node stuck in the DrainingNode stage.
3. On both nodes, verify available disk space: df -h /
   • Note: The Antrea image requires approximately 300 MB. Ensure at least 1 GB of free space is available to prevent partition exhaustion during the export/import process.

Note: It can be the case that the node are in a reboot cycle after they are repeatedly attempted to be made ready. In that case you can pause the cluster before proceeding the next steps:
kubectl patch cluster <cluster-name> --type merge -p '{"spec":{"paused": true}}'

Step 2: Export Image from a Healthy Node

1. SSH into the healthy worker node.
2. List the images to find the exact Antrea tag and SHA: 
   ctr -n k8s.io images list | grep -i antrea
3. Export the image to a temporary tar file: 
   ctr -n k8s.io images export /tmp/antrea.tar [FULL_IMAGE_REFERENCE] 
   Example: ctr -n k8s.io images export /tmp/antrea.tar localhost:5000/tkg/packages/core/antrea@sha256:740c8fb1d450b77cd4e4172b77e657f9b9433106d929222c22b82b50ab555638
4. Modify permissions to allow the transfer: 
   chown vmware-system-user:vmware-system-user /tmp/antrea.tar

Step 3: Import Image to the Impacted Node

1. SSH into the impacted (failing) worker node.
2. Copy the tar file from the healthy node:
   scp [HEALTHY_NODE_IP]:/tmp/antrea.tar /tmp/antrea.tar
3. Import the image into the container runtime: 
   ctr -n k8s.io images import /tmp/antrea.tar
4. Verify the Antrea agent pod transitions to a Running state. Once the CNI is active, the kubelet will successfully terminate the remaining pods, and the node drain will complete automatically.

Step 4: Cleanup

1. Remove the temporary tar files from both the healthy and impacted nodes:
    rm /tmp/antrea.tar

Step 5: Unpause the cluster.

1. If you had paused the cluster before Step 2, unpause it with:
   kubectl patch cluster <cluster-name> --type merge -p '{"spec":{"paused": false}}'

Additional Information

To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.