Pods stuck in Init state with Sandbox Errors(status 400)

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

When deploying workloads or Cloud-Native Network Functions (CNFs) on a VMware Telco Cloud Automation (TCA) cluster (e.g., version 3.3.0.1), pods fail to initialize and remain stuck in the init or ContainerCreating state.

Reviewing the pod events or kubelet logs reveals the following error:
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "...": plugin type="multus-shim" name="multus-cni-network" failed (add): CNI request failed with status 400

Environment

TCA 3.3.0.1
TCP 5.0.2 

Cause

The root cause is a target path definition error within the configuration parameters of the Istio CNI Node DaemonSet (istio-cni-node).

During the node bootstrap sequence, the install-cni container targets an internal Multus installation staging subdirectory instead of the active host-mapped directory (/etc/cni/net.d). Because the critical files istio-cni.conf and istio-cni.kubeconfig are missing from the standard directory, the multus-shim cannot validate the network namespace rules, resulting in a Status 400 failure.

Resolution

Permanent Fix

To permanently resolve this issue and ensure persistence across node re-imaging or scaling events, the resolution must be applied at the vendor application layer.

The vendor must update their Network Function (NF) deployment templates and installation scripts to target the standard active directory path.

1. Ammend CNF Files: Update the declarative templates for the Network Function to set the cni_conf_dir and kubeconfig_dir variables to /etc/cni/net.d.
   
   
2. Update CNF: Once the templates are corrected, update the CNF via TCA. This ensures that the automated node bootstrap sequence correctly writes the Istio CNI configuration and authentication files to the persistent host namespace at boot time.
   
   
3. Verify: Confirm that istio-cni.conf and istio-cni.kubeconfig are now being automatically placed in /etc/cni/net.d upon node creation.

Manual Workaround

For immediate restoration of service on existing nodes:

1. SSH into the affected worker node.
2. Manually copy istio-cni.conf and istio-cni.kubeconfig from the internal staging subdirectory into the active /etc/cni/net.d/ folder.
3. This will instantly unblock the container runtime and allow the pods to transition to a Ready state.

Note: Manual file transfers are non-persistent and will be wiped during node re-imaging, upgrades, or scaling events. The vendor template update is required for a permanent fix.