Error: "Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox"

Products

• VMware Telco Cloud Automation

Issue/Introduction

• Pods are not starting

• Pods reporting error:
  "Error adding container to network "k8s-pod-network": cannot allocate new block due to per host block limit"

• Below is the sample output of  ./calicoctl ipam check --show-problem-ips

Found 3356 IPs that are allocated in IPAM but not actually in use.
Scanning for IPs that are in use by a workload or node but not allocated in IPAM...
Found 0 in-use IPs that are not in active IP pools.
Found 0 in-use IPs that are in active IP pools but have no corresponding IPAM allocation.

Scanning for IPAM handles with no matching IPs...
Found 0 handles with no matching IPs (and 3365 handles with matches).
Scanning for IPs with missing handle...
Found 0 handles mentioned in blocks with no matching handle resource.
Check complete; found 3356 problems.

Environment

2.x, 3.x

Cause

Calico IPAM unable to allocate IP address to the PODs

Resolution

Release the unused IP Addresses

1. Lock the data store

   ./calicoctl datastore migrate lock

   Note: Sometimes, User might receive error as datastore not locked after executing the command. Please proceed with the next steps.

2. Generate a report using the check command

   ./calicoctl ipam check -o report.json

3. Release any unnecessary addresses

   ./calicoctl ipam release --from-report report.json

4. Unlock the data store

   ./calicoctl datastore migrate unlock

5. Restart the calico agents and calico-controller on the workload cluster.

6. Check the IPAM allocation and validate if the IP Problems are getting reduced.
./calicoctl ipam check --show-problem-ips (or) kubectl calico ipam check --show-problem-ips --allow-version-mismatch

7. Restart the worker nodes, if required.

Note: calicoctl is not available on the cluster controlplane by default. 

• Refer to the Install calicoctl page to install calicoctl on the control plane (SSH using VIP). Identify the Kubernetes version and use the appropriate calicoctl version.
• Refer to the projectcalico/calico page to identify the correct calicoctl version.

Example:

• K8s 1.24.10 is installed with calico v3.24.1
• Download the correct calicoctl version:
  curl -L https://github.com/projectcalico/calico/releases/download/v3.24.1/calicoctl-linux-amd64 -o calicoctl