Missing Secondary CNI Interfaces on Pods During Node Boot

Products

• VMware Telco Cloud Platform

Issue/Introduction

Application pods configured with secondary network annotations (using the k8s.v1.cni.cncf.io/networks key) fail to reach the Ready state (Ready: False) following a node boot, reboot, or rescheduling event. Although the container state displays Running with zero restarts, health and readiness checks continuously fail. The pod contains network annotations specifying secondary attachments, such as:

 

k8s.v1.cni.cncf.io/networks: <network-attachment-definition-1>, <network-attachmentdefinition-2>

 

Due to initialization timing during node startup, the pod container launches without its required secondary CNI network interfaces attached, leading to internal connectivity issues and probe failures.

Environment

TCP 5.1

Cause

A startup race condition occurs on the worker node between the application workload pods and the Multus CNI DaemonSet during host initialisation:

• Multus relies on copying plugin binaries and writing configuration files (e.g., /etc/cni/net.d/00- multus.conf) to the host filesystem upon node startup.
• If an application pod schedules and begins sandbox creation before Multus finishes populating the node CNI configuration, Kubelet falls back to standard single-interface network initialisation.
• Because Kubernetes CNI network attachments are generated exclusively during initial pod sandbox creation, affected pods remain missing their secondary interfaces indefinitely until recreated.

Resolution

Workaround:

To resolve this issue for affected pods, force Kubernetes to recreate the pod sandbox after the Multus CNI plugin is fully operational on the node.

Note:

Deleting the affected pod causes its parent controller (StatefulSet, Deployment, or ReplicaSet) to recreate it. During
recreation, Multus intercepts the network setup request and successfully attaches all configured secondary network
interfaces. 

 

Step 1: Delete the affected pod(s)

 

kubectl delete pod -n <namespace> <pod-name>

 

Step 2: Verification Steps

 

• Monitor the replacement pod until it achieves Ready: True: 

 

kubectl get pod -n <namespace> <pod-name> -w

 

• Inspect the pod details to confirm all secondary network interfaces are properly assigned: 

 

kubectl describe pod -n <namespace> <pod-name>