Error: adding container to network "ethName": error at storage engine: Could not allocate IP in range

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

Pod not in ready state after redeployment in tca. Error event found when describing the pod:

Warning FailedCreatePodSandBox 3m29s (x8866 over 3h58m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "################": plugin type="multus-cni-network" failed (add): [NameSpace/PodName/ethName]: error adding container to network "ethName": error at storage engine: Could not allocate IP in range: ip: #.#.#.#/ - #.#.#.# / range: net.IPNet{IP:net.IP{#,#,#,#}, Mask:net.IPMask{#,#,#,#}

 

Environment

TCA: 2.x, 3.1.1
TCP: 5.0

Cause

IP allocation issue with IP exhaustion in whereabouts

Resolution

Clear up IP allocations in whereabouts by following these steps:

1. Identify the other pods of the same statefulset/deployments:

   kubectl get pod -n namespace | grep -i <part-of-podname>

2. Identify if pod is deployment or statefulset:
   

   kubectl get deployment -n <namespace>
   
   if statefulset:
   
   kubectl get statefulset -n <namespace>

3. Depending on if pod is statefulset or deployment run the following:
   

   kubectl scale deployment -n <namespace> <name> --replicas=0
   
   if statefulset:
   
   kubectl scale statefulset -n <namespace> <name> --replicas=0
   

4. Run the command to identify all the pods in the same statefulset/deployments
   

   kubectl get overlappingrangeipreservations.whereabouts.cni.cncf.io -n whereabouts -o yaml | grep -C10 podname

5. Under each item in the array list take a look at the lines Name which is the ip address and podref which is the name of the pod and make a note of these. 
   • For statefulset pod the name of podref will be the same as the one that is having an issue.
   • For pods in a deployment it is possible that the name belongs to a pod that no longer exists.
6. Run the command and identify the ippool that the pod is using by matching the ip range from the error message that we got from the describe command on the pod  "Could not allocate IP in range: ip: #.#.#.#/ - #.#.#.#"
   

   kubectl get ippools.whereabouts.cni.cncf.io -A

7. Run the command to confirm if you see the name of the pods from the statefulset/deployment under allocation and podref
   

   kubectl get ippools.whereabouts.cni.cncf.io -n <namespace><name> -o yaml

8. Take a backup of this above yaml file
   

   kubectl get ippools.whereabouts.cni.cncf.io -n <namespace><name> -o yaml > ippool.yaml

9. Delete the ip of the pod that is having issues for statefulset or delete the pod that no longer exists for deployment pod which was obtained in step 4 by running 
   

   kubectl delete overlappingrangeipreservations.whereabouts.cni.cncf.io -n <namespace> #.#.#.#

10. Delete the ippools.whereabouts.cni.cncf.io and delete the number,id,podref for the pod with the issue under allocations by running
    

    kubectl delete ippools.whereabouts.cni.cncf.io -n <namespace><name>

11. Restart the daemonsets for whereabouts and multus:
    

    kubectl rollout restart daemonset kube-multus-ds -n kube-system
    kubectl rollout restart daemonset whereabouts -n kube-system

12. Wait for the pod to get ip allocated
    

    watch -n 1 'kubectl get pods -A | grep <part of podname for statefulset/deployment>'

If you have followed the steps above and it did not resolve the issue, validate the network-attachment-definitions to see if the ip exclusion is setup properly

1. From the control plane run
   

   kubectl get network-attachment-definitions -n namespace NADname -o yaml

2.  Search for exclude and validate if the CIDR and the ip exclude range is configured properly
   

   "exclude": ["#.#.#.#/#","#.#.#.#/#"]