Pod Sandbox Creation Fails with Multus CNI 'Link not found' Error Due to Uppercase Network Interface Name

Products

VMware Telco Cloud Automation

Issue/Introduction

When deploying Cloud-Native Network Functions (CNFs) or standard workloads in VMware TCA 3.4.0.1, pods fail to initialise and remain stuck in a ContainerCreating or Error state.

Error Message

Running kubectl describe pod <pod-name> or inspecting the kubelet logs reveals a FailedCreatePodSandBox event containing a Multus CNI failure:

Warning  FailedCreatePodSandBox  99s (x7100 over 15h)   kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2578c9fbc4fc32b8b48584666a350638a6638b65390e053f571bc5236f39c595": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"..." Netns:"..." IfName:"eth0" Args:"..." ERRORED: error configuring pod [cbf/podname-xxxx-0] networking: [cnf-podname-0/ac33fb31-1293-488d-a0ae-09fa75a23d4b:ipvlan-nw-oam-ip-xxxx-xxx]: error adding container to network "ipvlan-nw-oam-ip-xxxx-xxx": failed to lookup master "OAM_External": Link not found ': StdinData: {"capabilities":{"portMappings":true},"clusterNetwork":"/host/etc/cni/net.d/10-calico.conflist","cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","type":"multus-shim"}

Environment

3.4.0.1

Cause

This issue is caused by a case-sensitivity mismatch between the TCA UI configuration and the underlying node automation:

TCA UI Entry: When defining network interfaces under NodePool > Edit > Node policy > Network Interface Name, the UI allows the entry of uppercase or mixed-case string labels (e.g., OAM_External).
Underlay Worker Node Processing: The system automation automatically normalizes and creates these underlying network link interfaces entirely in lowercase on the actual worker node.
CNI Execution: During CNF deployment, Kubernetes and Multus expect an exact, case-sensitive string match. Because the CNF deployment refers to the uppercase label, the CNI tries to find a master link named "OAM_External". Since the OS only has it registered in lowercase, the lookup fails with a Link not found error.

Resolution

To resolve the problem, ensure that the network adapter label names are normalised to strictly lowercase characters in both the TCA UI configuration and the CNF deployment configurations.

Step 1: Correct the TCA NodePool Network Policy

Log in to the Telco Cloud Automation UI.
Navigate to your cluster infrastructure configuration and locate the affected NodePool.
Go to Node policy > Network Adapter > Network Interface Name
Modify any mixed-case or uppercase network labels to be strictly lowercase (e.g., change OAM_External to oam_external).
Save and apply the configuration changes to update the NodePool.

Step 2: Update the CNF Deployment YAML

The workload configuration must align with the lowercase formatting utilized at the infrastructure layer.

Open your CNF deployment manifest file (e.g., cnf-deployment.yaml or pod.yaml).
Locate the network annotations metadata block (e.g., k8s.v1.cni.cncf.io/networks).
Convert the network references/labels to lowercase to match Step 1.

Configuration Comparison:
Incorrect (Mixed/Upper Case): k8s.v1.cni.cncf.io/networks: ipvlan-nw-OAM_External
Correct (All Lowercase): k8s.v1.cni.cncf.io/networks: ipvlan-nw-oam_external

Apply the corrected YAML file to the cluster: kubectl apply -f cnf-deployment.yaml

kubectl apply -f cnf-deployment.yaml

Step 3: Verification

Clean up any stale or stuck pods remaining from the failed deployment attempt to force a fresh reconciliation:
```
kubectl delete pod <stuck-pod-name> -n <namespace>
```
Monitor the newly generated pods to ensure they successfully provision the network sandbox and transition to a Running status:
```
kubectl get pods -n <namespace> -w
```