Pod IP Block expansion in TKGi

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• Pod IP Block exhaustion can occur when there're no more available Subnets associated to the Pod IP Block CIDR configured for the TKGi cluster.
  
  
• Every time a Namespace is created in a cluster, a new Subnet within the Pod IP Block CIDR will be created for that Namespace.
  If a Subnet is exhausted and you create a pod in the corresponding Namespace, NCP will automatically create a new Subnet and assign an IP from it to the pod. If the pod is deleted, then this extra Subnet will get deleted too (if no more pods exist in the new Namespace).

 

Environment

1.24.3

Cause

Too many Namespaces in the cluster, exhausting the available Subnets within the Pod IP Block associated to the cluster.

Resolution

1. To expand the Pod IP Block in a TKGi cluster, we can apply a network-profile to the cluster and configure the pod_ip_block_ids field.
   
   
2. Follow the steps outlined in the Docs Add Pod IPs.
   
   
3. You should always include the pod_ip_block_id the cluster is already using in the array. For example:
   
   

   {
       "description": "Example network profile for adding pod IP addresses to an existing cluster",
       "name": "pod-ips-add",
       "parameters": {
         "pod_ip_block_ids": [
           "<existing_pod_ip_block_ids>",
           "<new_pod_ip_block_ids>"
         ]
       }
   }
   

   
   
   
4. You can reorder the pod_ip_block_id list and input a newly created Pod IP Block on NSX first, so from now on it will be used by NCP with preference over the old one to assign IP addresses:
   
   

   {
       "description": "Example network profile for adding pod IP addresses to an existing cluster",
       "name": "pod-ips-add",
       "parameters": {
         "pod_ip_block_ids": [
           "<new_pod_ip_block_ids>",
           "<existing_pod_ip_block_ids>"
         ]
       }
   }

   
   
   
5. You cannot change a cluster’s network profile to remove Pod IP Block IDs.

Additional Information

General information on Pod IP Block can be found in the following references:

Pods IP Block

TKGI with NSX-T: Quick guide with examples on pod_ip_block_ids, node_ip_block_ids, pod_subnet_prefix, node_subnet_prefix parameters