TKGI with NSX-T: Quick guide with examples on pod_ip_block_ids, node_ip_block_ids, pod_subnet_prefix, node_subnet_prefix parameters

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware Tanzu Kubernetes Grid Integrated Edition (Core)
• VMware Tanzu Kubernetes Grid Integrated Edition 1.x
• VMware Tanzu Kubernetes Grid Integrated EditionStarter Pack (Core)

Issue/Introduction

This article provides answers to a few common questions on the following parameters used within Tanzu Kubernetes Grid Integrated Edition with some examples:

pod_ip_block_ids,

node_ip_block_ids,

pod_subnet_prefix,

node_subnet_prefix

Environment

Tanzu Kubernetes Grid Integrated Edition

NSX Data Center

Resolution

Q: Is our Node IP Block appropriately sized for existing and future clusters?

Answer: Broadcom Support cannot provide infrastructure sizing and planning guidance. However, let's walk through two example Node IP Blocks to show what it will give you  

NOTE:

• • Every Kubernetes cluster deployed by TKGI owns a /24 subnet (which is equal to the default value of node_subnet_prefix=24)
    
    
  • The recommended size of each NODE IP BLOCK is /16.
    
    
  • Starting in TKGI 1.16.0, you can expand new node subnets by adding Node IP Block IDs to a Network Profile
    
    
  • TKGI >= 1.16.0: Supports adding new node subnets (NODE IP BLOCKs) to an existing Network Profile, Node Network IP Block
    
    
  • Refer to the release TKGI release notes.

 

Example1:

Node IP Block: 10.218.128.0/16 which carves out Node Subnets with /24 per kubernetes Cluster

Total Clusters and Nodes possible using the default /24 subnet:

Subnets (number of TOTAL Clusters) =   256

Total Hosts (total number of cluster nodes) = 65024

Maximum kubernetes nodes per cluster: (65024 / 256) = 254

 

Example2:

Node IP Block: 10.218.128.0/18 which carves out Node Subnets with /24 per kubernetes Cluster

Total Clusters and Nodes possible using the default /24 subnet:

Subnets (number of TOTAL Clusters) =   64

Total Hosts (total number of cluster nodes) = 16256

Maximum kubernetes nodes per cluster: (16256 / 64) = 254

  

Refer to the following guides for more details:

Network Planning for Installing Tanzu Kubernetes Grid Integrated Edition with VMware NSX

Which discusses:

Nodes IP Block

Customize Node Networks

 

 

 

Q: Is my Pod IP Block appropriately sized for existing and future cluster workloads?

Answer: Broadcom Support cannot provide infrastructure sizing and planning guidance. However, let's walk through two example Pod IP Blocks to show what it will give you  

NOTE:

• • The recommended size of each POD IP BLOCK is /16.
    
    
  • Typically there are 7 default Namespaces for kubernetes cluster.  It can be important to take those into account during your own planning.

 

Example1:

Pod IP Block: 10.218.128.0/16 which carves out Subnets with /24 for each kubernetes Namespace (across all clusters)

Total Namespaces and Pods across all Clusters using recommended /16 subnet

Maximum Subnets (Total number of Namespaces across ALL clusters) = 256

Total Hosts (number of Pods across all clusters) = 65024

Maximum Pods per Namespace: (65024 /256) = 254

IMPORTANT:  If the number of pods in a Namespace goes beyond 254, then an additional Subnet will be carved out of the POD IP BLOCK range and used by the same Namespace.  Thus, only 255 subnets would remain.  The number of potential Namespaces would be lowered to 255 (from 256 max subnets)

 

Example2:

Pod IP Block: 10.218.128.0/18 which carves out Subnets with /24 for each kubernetes Namespace (across all clusters)

Total Namespaces and Pods across all Clusters using recommended /16 subnet

Maximum Subnets (Total number of Namespaces across ALL clusters) = 64

Total Hosts (number of Pods across all clusters) = 16256

Maximum Pods per Namespace: (65024 /256) = 254

IMPORTANT:  If the number of pods in a Namespace goes beyond 254, then an additional Subnet will be carved out of the POD IP BLOCK range and used by the same Namespace.  Thus, only 255 subnets would remain.  The number of potential Namespaces would be lowered to 63 (from 64 max subnets)

 

Refer to the following guides for more details:

Network Planning for Installing Tanzu Kubernetes Grid Integrated Edition with VMware NSX

Which discusses:

Pods IP Block

 

 

 

Q: How do the node_subnet_prefix, pod_subnet_prefix parameters affect the default values of the Nodes IP Block and Pods IP Block?

NOTE:

Nodes IP Block (node_ip_block_ids)  is Updatable starting with TKGI 1.16.0

Pods IP Block (pod_ip_block_ids) is also Updatable

 

Info about node_subnet_prefix:

• • Default: "node_subnet_prefix" = 24,
    
    
  • Relationship to NODE IP BLOCKS (and the node_ip_block_ids parameter):
    
    
    • This value is also used as the Default value for node_ip_block_ids.
      
      
    • Each Kubernetes cluster deployed by Tanzu Kubernetes Grid Integrated Edition owns a /24 subnet.
      
      
    • This is referenced by the Default of node_subnet_prefix of /24
      
      Example:
      
      
      • By changing the value of node_subnet_prefix to /22
        
        
      • Node IP Block: 10.218.128.0/16 would carves out Node Subnets with /22 per kubernetes Cluster
        
        
      • Subnets (maximum number of Clusters) = 64
        
        
      • Total Hosts (total number of cluster Nodes) = 65408
        
        
      • Maximum kubernetes nodes per cluster: (65408 / 64) = 1022
        
        

Refer to the Reference on Node Subnet Prefix

 

Info about pod_subnet_prefix:

• • Default: "pod_subnet_prefix" = 24,
    
    
  • Relationship to POD IP BLOCKS (and the pod_ip_blocks_ids parameter):
    
    
    • This value is also used as the Default value for pod_ip_block_ids.
      
      
    • Each time a Kubernetes namespace is created, a subnet from the pods IP block is allocated. The default size of the subnet carved from this block for such purposes is /24(default pod_subnet_prefix)

Example:

• • • • By changing the value of pod_subnet_prefix to /22
        
        
      • Node IP Block: 10.218.128.0/16 would carves out Namespace Subnets with /22 to be used across all TKGI Clusters
        
        
      • Subnets (maximum number of Namespaces) = 64
        
        
      • Total Hosts (total number of Pods across all Clusters) = 65408
        
        
      • Maximum kubernetes Pods per Namespace: (65408 / 64) = 1022
        
        

Refer to the Reference on Pod Subnet Prefix

 

 

 

Q: Is my Floating IP Pool sufficient for the Kubernetes Services and cluster Load Balancers in my environment?

Answer: Floating IPs (FIPs) in TKGI when NSX Data Center is used:

• • By the Kubernetes Load Balancer services of all clusters
    
    
  • By the Kubernetes Master nodes IP(s) of all clusters
    
    
  • As SNAT IP addresses, whenever any Namespace is created (when in NAT mode) in any cluster

Answer: You can always add more Floating IP Pools if your default IP Block is exhausted or you are reaching its limit through a Network Profile 

 

 

Q: Can I expand the number of Floating IPs in my environment?

Answer: Yes

 

 

Q: How do I expand available Floating IPs?

Answer:

During TKGI deployment, a single Floating IP Pool ID is populated by the TKGI Operator into the Opsmanager -> TKGI tile -> Networking tab

That Floating IP Pool ID becomes the default Floating Pool ID used by all Kubernetes clusters

When deployed NSX Data Center, an operator can create more Floating IP Pool IDs and expand through a Network Profile as needed 

Steps:

• • If you run into FIP IP Exhaustion or want to expand, refer to the KB:

Add new Floating IPs (FIPs) to TKGi

• • If the plan is to update an existing Network Profile, it is important to understand:

Update-Cluster Network Profile Validation Rules

 

NOTE:

fip_pool_ids is an array used within a Network Profile.  It is Updatable. You can create additional FIP IDs to expand your Floating IP pool through a Network Profile

Not all Network Profile parameters are Updatable.  Refer to Network Profile Parameters

•  

 

 

 

Q: Are there any adjustments or recommendations to improve this setup based on TKGI and NSX-T best practices?

Answer:

Broadcom Support is not able to give architectural planning guidance specific to customer business plans.  If you need architectural and related planning, customer account teams will help you with a PSO engagement to assist. 

Broadcom support recommends the following planning guides and topics: 

Network Planning for Installing Tanzu Kubernetes Grid Integrated Edition with VMware NSX

 

NSX Deployment Topologies for Tanzu Kubernetes Grid Integrated Edition

 

Considerations for Using the VMware NSX Policy API with TKGI

 

Network Profile Parameters

-> Top-Level Parameters

-> cni_configurations Parameters

 

Using Network Profiles (NSX Only)

 

Creating and Managing Network Profiles (NSX Only)

-> Network Profile Example

 

Customizing Pod Networks