CNI Antrea pods in CrashLoopBackOff due to unreachable kube-apiserver

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

Due to the disruption of inter-pod communication, the Kubernetes cluster is non-functional.

• antrea-agent and antrea-controller pods are CrashLoopBackOff
• coredns pods are also not up

kubectl get pods -A

NAME                                 READY   STATUS              RESTARTS
antrea-agent-6n96r                   1/2     CrashLoopBackOff    27 (2m ago)
antrea-agent-7r5bc                   1/2     CrashLoopBackOff    27 (2m31s ago)
antrea-agent-7vhz2                   1/2     CrashLoopBackOff    27 (2m29s ago)
antrea-controller-6cd4fd947c-sj2lc   0/1     CrashLoopBackOff    24 (2m37s ago)
coredns-5fb8b9b9f4-xb6qq             0/1     Unknown             0
coredns-6689c69bcd-hc75r             0/1     ContainerCreating   0
coredns-6689c69bcd-qvbdt             0/1     ContainerCreating   0

 

antrea-controller can not reach the kube-apiserver (https://##.##.#.#:443).

kubectl -n kube-system logs antrea-controller

F*** HH:MM:SS.sss       1 main.go:53] Error running controller: error creating API server config: unable to load configmap based request-header-client-ca-file: Get "https://##.##.#.#:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp ##.##.#.#:443: connect: network is unreachable

 

antrea-agent is not up because the initialization process has failed.

kubectl -n kube-system logs antrea-agent

E*** HH:MM:SS.sss       1 agent.go:925] "Spec.PodCIDR is empty for Node. Please make sure --allocate-node-cidrs is enabled for kube-controller-manager and --cluster-cidr specifies a sufficient CIDR range" err="failed to get Node with name <NODE_NAME> from K8s: Get \"https://##.##.#.#:443/api/v1/nodes/<NODE_NAME>\": dial tcp ##.##.#.#:443: connect: network is unreachable" nodeName="<NODE_NAME>"
F*** HH:MM:SS.sss      1 main.go:53] Error running agent: error initializing agent: CIDR string is empty for Node <NODE_NAME>

Environment

Tanzu Kubernetes Grid Management

Cause

This issue occurs when the Kubernetes node does not have a default route configured.

Example scenario: Misconfiguration on the DHCP server responsible for distributing the default route.

• Github - K8s can't live without a default route
• Github - kubernetes always requires a default route

Resolution

Ensure that all Kubernetes nodes have the correct default route configured (Review the DHCP server configuration).