Error: "Istio pods are failing due to error " plugin type="istio-cni" name="istio-cni" failed (add): exit status 2"

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Pods are in 'init' state and it is failed to go 'running' state
• we see below error when we describe 'init' pods:
  

  Warning  FailedCreatePodSandBox  3m13s (x2242 over 8h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "12345": plugin type="multus" name="multus-cni-network" failed (add): [example/example-12345:k8s-pod-network]: error adding container to network "k8s-pod-network": Calico is currently not ready to process requests

   

Environment

3.x

Cause

Calico's datastore Ready will be marked as false and that is a way to lock datastore by calico controller to stop ip allocation or writing other variables when k8s api server is not reachable for longer time. 

Resolution

• Run below command to fetch Calico's datastore ready status:
  

  kubectl get clusterinformations -o yaml

• The output will be similar to below:
  

  apiVersion: v1
  items:
  apiVersion: crd.projectcalico.org/v1
  kind: ClusterInformation
  metadata:
  annotations:
  projectcalico.org/metadata: '
  {"uid":"###-##","creationTimestamp":"YYYY-MM-DDTHH:MM:SSZ"}
  'creationTimestamp: "YYYY-MM-DDTHH:MM:SSZ"
  generation: 4
  name: default
  resourceVersion: "####"
  uid: ###-###
  spec:
  calicoVersion: v3.24.1
  clusterGUID: ###-###
  clusterType: k8s,bgp,kubeadm,kdd
  datastoreReady: false
  kind: List
  metadata:
  resourceVersion: ""

• If we could see 'datastoreReady' is showing as 'false' in above output which is causing the problem.
• So, change it's value to 'true' and then restart the pods , that should change pods status to running.