PODs failed with error "Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox"

Products

• VMware Tanzu Kubernetes Grid Service (TKGs)
• VMware Telco Cloud Automation

Issue/Introduction

User encountered below errors:

- Error adding container to network "k8s-pod-network": cannot allocate new block due to per host block limit 
- Followed KB article Pods not coming in ready state due to IP allocation issues using whereabouts 
- Below is the sample output of  calicoctl ipam check --show-problem-ips

Found 3356 IPs that are allocated in IPAM but not actually in use.
Scanning for IPs that are in use by a workload or node but not allocated in IPAM...
Found 0 in-use IPs that are not in active IP pools.
Found 0 in-use IPs that are in active IP pools but have no corresponding IPAM allocation.

Scanning for IPAM handles with no matching IPs...
Found 0 handles with no matching IPs (and 3365 handles with matches).
Scanning for IPs with missing handle...
Found 0 handles mentioned in blocks with no matching handle resource.
Check complete; found 3356 problems.

 

Environment

_{TCA 2.X}

_{TKG 2.X}

Cause

_{calico ipam unable to release IP address to the PODs} 

Resolution

Steps to release the unused IP Address

1. Lock the data store
   

   calicoctl datastore migrate lock

   Note: Sometimes, User might receive error as datastore not locked after executing the command. Please proceed with the next steps.
   
   
2. Generate a report using the check command
   

   calicoctl ipam check -o report.json

3. Release any unnecessary addresses
   

   calicoctl ipam release --from-report report.json

4. Unlock the data store
   

   calicoctl datastore migrate unlock

   
   
   

Additional Information

<sub>- Restart the calico agents and calico-controller on the workload cluster.
- Check the IPAM allocation and validate if the IP Problems are getting reduced.</sub>

calicoctl ipam check --show-problem-ips (or) kubectl calico ipam check --show-problem-ips --allow-version-mismatch

_{- Restart the worker nodes, if required}