Pod not in ready state. Looking at the pod events we can see the following:
Warning FailedCreatePodSandBox 3m29s (x8866 over 3h58m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "################": plugin type="multus-cni-network" failed (add): [NameSpace/PodName/ethName]: error adding container to network "ethName": error at storage engine: Could not allocate IP in range: ip: 192.168.1.10/ - 192.168.1.15 / range: net.IPNet{IP:net.IP{0x#,0x#,0x#,0x#}, Mask:net.IPMask{0x#,0x#,0x#,0x#}
2.x
The error is talking about IP allocation issue which points to IP exhaustion in whereabouts.
In order to fix this issue we need to manually clear up IP allocations in whereabouts by following these steps:
kubectl get pod -n namespace | grep -i podname
kubectl get overlappingrangeipreservations.whereabouts.cni.cncf.io -n whereabouts -o yaml | grep -C10 podname
kubectl get ippools.whereabouts.cni.cncf.io -n whereabouts
kubectl get ippools.whereabouts.cni.cncf.io -n whereabouts 192.168.1.1-24 -o yaml
kubectl get ippools.whereabouts.cni.cncf.io -n whereabouts 192.168.1.1-24 -o yaml > ippool.yaml
kubectl delete overlappingrangeipreservations.whereabouts.cni.cncf.io -n whereabouts 192.168.1.10
kubectl edit ippools.whereabouts.cni.cncf.io -n whereabouts 192.168.1.1-24
If you have followed the steps above and it did not resolve the issue, validate the network-attachment-definitions to see if the ip exclusion is setup properly
kubectl get network-attachment-definitions -n namespace NADname -o yaml
"exclude": ["192.168.1.10/30","192.168.1.118/30"]