Pods not coming in ready state due to IP allocation issues using whereabouts

Products

• VMware Telco Cloud Automation

Issue/Introduction

Looking at the pod events we can see the following:

Warning FailedCreatePodSandBox 3m29s (x8866 over 3h58m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "################": plugin type="multus-cni-network" failed (add): [NameSpace/PodName/ethName]: error adding container to network "ethName": error at storage engine: Could not allocate IP in range: ip: 192.168.1.10/ - 192.168.1.15 / range: net.IPNet{IP:net.IP{0x#,0x#,0x#,0x#}, Mask:net.IPMask{0x#,0x#,0x#,0x#}

The error is talking about IP allocation issue which points to IP exhaustion in whereabouts.

Environment

2.x

Resolution

In order to fix this issue we need to manually clear up IP allocations in whereabouts by following these steps:

1. Identify the other pods of the same statefulset/deployments by running 

   kubectl get pod -n namespace | grep -i podname

2. Make a note of the pods that are part of the same statefulset/deployments
3. Run the command to identify all the pods in the same statefulset/deployments
   

   kubectl get overlappingrangeipreservations.whereabouts.cni.cncf.io -n whereabouts -o yaml | grep -C10 podname

4. Under each item in the array list take a look at the lines Name which is the ip address and podref which is the name of the pod and make a note of these. Note: For statefulset pod the name of podref will be the same as the one that is having an issue. For pods in a deployment it is possible that the name belongs to a pod that no longer exists.
5. Run the command and identify the ippool that the pod is using by matching the ip range from the error message that we got from the describe command on the pod  "Could not allocate IP in range: ip: 192.168.1.10/ - 192.168.1.15"
   

   kubectl get ippools.whereabouts.cni.cncf.io -n whereabouts

6. Run the command to confirm if you see the name of the pods from the statefulset/deployment under allocation and podref
   

   kubectl get ippools.whereabouts.cni.cncf.io -n whereabouts 192.168.1.1-24 -o yaml

7. Take a backup of this above yaml file
   

   kubectl get ippools.whereabouts.cni.cncf.io -n whereabouts 192.168.1.1-24 -o yaml > ippool.yaml

8. Delete the ip of the pod that is having issues for statefulset or delete the pod that no longer exists for deployment pod which was obtained in step 4 by running 
   

   kubectl delete overlappingrangeipreservations.whereabouts.cni.cncf.io -n whereabouts 192.168.1.10

9. Edit the ippools.whereabouts.cni.cncf.io and delete the number,id,podref for the pod with the issue under allocations by running
   

   kubectl edit ippools.whereabouts.cni.cncf.io -n whereabouts 192.168.1.1-24

    
10. Wait for the pod to get ip allocated