Failed to add 2nd interface to pod because of expired token in multus kubeconfig

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

• When creating a pod with annotations to add a second NIC to the pod using multus (with or without whereabouts), multus pod will show ‘Unauthorized’ error.
  
  2024-04-17T03:30:44Z [error] SetNetworkStatus: failed to query the pod POD_NAME-cluster-0 in out of cluster comm: Unauthorized
2024-04-17T03:30:44Z [error] Multus: error unsetting the networks status: SetNetworkStatus: failed to query the pod POD_NAME-cluster-0 in out of cluster comm: Unauthorized
2024-04-17T03:30:55Z [error] delegateAdd: pod nil pointer: namespace: NAMESPACE_NAME, name: POD_NAME-cluster-0, container id: CONTAINER_ID, pod: nil
  
  
• To turn on the multus log before TKG 2.3 update the Node's ‘/etc/cni/net.d/00-multus.conf’ file. Add the following:
  
  "logLevel": "debug",
"logFile": "/var/log/multus.log"

  
  
• Reference the Multus-cni documentation for formatting examples as needed.

Cause

• Before TKG 2.3, multus will only create a kubeconfig at starting time. It won’t rotate the certificate and token inside its kubeconfig if they expire.
• In all TKG versions, whereabouts will only create kubeconfig at starting time. It won’t rotate the certificate and token inside its kubeconfig  if they expire.

Resolution

Multus 4.0.1, which is included in TKG 2.3, has a mechanism to rotate the certificate and token. Whereabouts doesn’t have this mechanism to rotate the certificate and token as of TKG 2.5.

 

 

Notes Prior to Performing Workaround

• The restarting of multus pod and whereabouts pod (if it’s deployed) shouldn’t cause issues to existing workload pods.
• However, if you are creating new workload pods that require a second interface, while multus and whereabouts pods are rolling out, these new workload pods will likely miss the second interface.
• It is therefore recommend to pause other actions when rolling out multus and whereabouts pods.

 

Use the below workaround to correct the tokens in multus and whearabouts pods:

 

1. Recreate(kill) multus pods or whereabouts pods if their tokens are about to expire. The restart should ideally be done preventatively before the token in `multus.kubeconfig` and `whereabouts.kubeconfig` expire. Users can inspect the file and decode the token to check expiry date. 
   
   Example to check tokens of multus daemonset:
   
   

   for p in $(kubectl get pod -n kube-system -l app=multus --no-headers -o name); do echo $p kubeconf-token exp=$(date -d @$(kubectl exec -it -n kube-system $p -- cat /host/etc/cni/net.d/multus.d/multus.kubeconfig | grep token | awk '{ print $2 }' | awk -F'.' '{print $2}' | tr '_-' '/+' |\
   sed 's/\./\n/g' | \
   sed 's/-/+/g; s/_/\//g' | \
   while read part; do \
       len=$((${#part} % 4)); \
       if [ $len -eq 2 ]; then part="${part}=="; fi; \
       if [ $len -eq 3 ]; then part="${part}="; fi; \
       echo "$part" | base64 -d; \
   done  | jq '.exp')); done 

   
   
   
   Example to check all pods in the whereabouts daemonSet, if whereabouts is installed:
   
   

   for p in $(kubectl get pod -n kube-system -l app=whereabouts --no-headers -o name); do echo $p kubeconf-token exp=$(date -d @$(kubectl exec -it -n kube-system $p -- cat /host/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig | grep token | awk '{ print $2 }' | awk -F'.' '{print $2}' | tr '_-' '/+' |\
   sed 's/\./\n/g' | \
   sed 's/-/+/g; s/_/\//g' | \
   while read part; do \
       len=$((${#part} % 4)); \
       if [ $len -eq 2 ]; then part="${part}=="; fi; \
       if [ $len -eq 3 ]; then part="${part}="; fi; \
       echo "$part" | base64 -d; \
   done  | jq '.exp')); done

   
   
   
   
2. If the tokens expire and cause the newly created workload pods to lose the expected second interface, recreate workload pods that are missing the second interface after multus and whereabouts pods are back to running
   
   Example command to perform the rolling restart.
   
   Multus:
   
   kubectl -n kube-system rollout restart daemonset --selector=app=multus
   
   
   Whereabouts:
   
   kubectl -n kube-system rollout restart daemonset --selector=app=whereabouts
   
   
   
   
3. To monitor the rollout of pods
   
   Multus
   
   watch kubectl -n kube-system get pod --selector=app=multus
   
   
   Whereabouts
   
   watch kubectl -n kube-system get pod --selector=app=whereabouts