Calico node reports unauthorized error after token expires, if TKR < 1.24

Products

• VMware
• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

Symptoms:

• If the environment is configured using TKR < 1.24 version, regardless of TKGm or TKGs, Calico version is 3.19, and by default it doesn’t have CALICO_MANAGE_CNI set, so it won’t automatically refresh the token once the token used for Calico will expire. 
• The Calico-node Pod will be failing to start or be deleted. In Calico-node pod log, it will show Unauthorized like below:

 

Failing pods will report the following error:

Warning FailedCreatePodSandBox 68s (x465 over 104m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e0641458c27a65572308c5367615cd90a2f8d277b3874cd7bc725d3b45b2e9e1": [namespace/cluster-name-log-transformer-7b9b47b986-lfrb6:k8s-pod-network]: error adding container to network "k8s-pod-network": error getting ClusterInformation: connection is unauthorized: Unauthorized.

Environment

VMware vSphere 7.0 with Tanzu
VMware Tanzu Kubernetes Grid 1.x
VMware vSphere 8.0 with Tanzu

Cause

Calico version 3.19, by default doesn’t have the CALICO_MANAGE_CNI variable set, so it won’t automatically refresh the token once the token used for Calico expires.

Resolution

In TKr 1.24, Calico will be upgraded to 3.24, and it will turn on CALICO_MANAGE_CNI by default.

 

Workaround:

Restarting the calico-node-ID pod will resolve this issue. Restarting the Calico-node pod will trigger the install-calico initContainer to regenerate the token.