Kubernetes kubelet shows 'no IP addresses available in network' errors

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:
When running various BOSH errands against a k8s cluster, the kubelet logs, /var/vcap/sys/log/kubelet/kubelet.stderr.log, on a worker node shows these errors:

cni.go:259] Error adding network: no IP addresses available in network: flannel-network
cni.go:227] Error while adding to cni network: no IP addresses available in network: flannel-network

And one or more pods will be in Pending status like below:

$ kubectl get pods -o wide

NAME                          READY     STATUS    RESTARTS   AGE       IP        NODE
kuard-test-64f5d959cc-86qfm   0/1       Pending   0          1d        <none>    <none>
kuard-test-64f5d959cc-b59nr   0/1       Pending   0          1d        <none>    <none>
kuard-test-64f5d959cc-fkcmj   0/1       Pending   0          1d        <none>    <none>
kuard-test-64f5d959cc-tcdr5   0/1       Pending   0          1d        <none>    <none>

Environment

Cause

The following chronology of events can cause this:

1: A Kubernetes Worker Node becomes unavailable for some reason (crashes, offline, etc.).

2: That worker node is unavailable for 24 hours or longer, then brought back online via "BOSH recreate".

3: k8s pods will then show as stuck in state, such as ContainerCreating.

Example kubectl output:

$ kubectl get pods -o wide
NAME                          READY     STATUS              RESTARTS   AGE       IP             NODE
kuard-test-64f5d959cc-7w6mg   0/1       ContainerCreating   0          8s        <none>         fc717f51-ee74-4e8f-9b13-7fa8ca3016d0

Snippet of event from the kubelet.stderr.log on the worker node shows:

network: "cni0" already has an IP address different from 10.200.##.#/24

The above can be resolved with this article:
PKS Flannel network gets out of sync with docker bridge network (cni0)

4: However, the above the subnet mismatch between flannel.1 and cni0 is unresolved and further BOSH errands are executed against the cluster, it can cause the error no IP addresses available in network on the worker node.

Example event from kubelet.stderr.log on worker VM:

E0815 16:45:02.382364    8014 cni.go:259] Error adding network: no IP addresses available in network: flannel-network
E0815 16:45:02.382392    8014 cni.go:227] Error while adding to cni network: no IP addresses available in network: flannel-network

Which was caused by /var/lib/cni/networks/flannel-network directory to be filled with flannel-network files named 10.200.##.##.  

Resolution

Move aside or remove the stale/unused address files shown in/var/lib/cni/networks/flannel-network:

1. bosh ssh -d <deployment_name> worker
2. cd /var/lib/cni/networks/flannel-network
3. Remove or move aside the unused 10.200.##.## files

Then resolve the Flannel network mismatch issue shown earlier.