TKGI with NSX-T: Quick guide with examples on pod_ip_block_ids, node_ip_block_ids, pod_subnet_prefix, node_subnet_prefix parameters
search cancel

TKGI with NSX-T: Quick guide with examples on pod_ip_block_ids, node_ip_block_ids, pod_subnet_prefix, node_subnet_prefix parameters

book

Article ID: 387102

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated Edition VMware Tanzu Kubernetes Grid Integrated (TKGi) VMware Tanzu Kubernetes Grid Integrated Edition (Core) VMware Tanzu Kubernetes Grid Integrated Edition 1.x VMware Tanzu Kubernetes Grid Integrated EditionStarter Pack (Core)

Issue/Introduction

This article provides answers to a few common questions on the following parameters used within Tanzu Kubernetes Grid Integrated Edition with some examples:

pod_ip_block_ids,

node_ip_block_ids,

pod_subnet_prefix,

node_subnet_prefix

Environment

Tanzu Kubernetes Grid Integrated Edition

NSX Data Center

Resolution

Q: Is our Node IP Block appropriately sized for existing and future clusters?

Answer: Broadcom Support cannot provide infrastructure sizing and planning guidance. However, let's walk through two example Node IP Blocks to show what it will give you  

NOTE:

    • Every Kubernetes cluster deployed by TKGI owns a /24 subnet (which is equal to the default value of node_subnet_prefix=24)

    • The recommended size of each NODE IP BLOCK is /16.

    • Starting in TKGI 1.16.0, you can expand new node subnets by adding Node IP Block IDs to a Network Profile

    • TKGI >= 1.16.0: Supports adding new node subnets (NODE IP BLOCKs) to an existing Network Profile, Node Network IP Block

    • Refer to the release TKGI release notes.

 

Example1:

Node IP Block: 10.218.0.0/16 which carves out Node Subnets with /24 per kubernetes Cluster

Total Clusters and Nodes possible using the default /24 subnet:

      • Maximum kubernetes nodes per cluster = 253
      • Subnets (number of TOTAL Clusters): 256
      • Total Hosts (total number of cluster nodes): (253 * 256) = 64768

Note:  NCP/NSX won't assign three of the IP addresses to any node/pod.

For example, in a /24 Subnet there're 256 theoretical available IP addresses. NCP/NSX won't assign x.x.x.0 (Network address), x.x.x.1 (Gateway address), and x.x.x.255 (Broadcast address) to any node/pod, so in practical terms there'll be only 253 available IP addresses per Subnet.

Example2:

Node IP Block: 10.218.128.0/18 which carves out Node Subnets with /24 per kubernetes Cluster

Total Clusters and Nodes possible using the default /24 subnet:

      • Maximum kubernetes nodes per cluster = 253
      • Subnets (number of TOTAL Clusters): 64
      • Total Hosts (total number of cluster nodes): (253 * 64) = 16192

Note:  NCP/NSX won't assign three of the IP addresses to any node/pod.

For example, in a /24 Subnet there're 256 theoretical available IP addresses. NCP/NSX won't assign x.x.x.0 (Network address), x.x.x.1 (Gateway address), and x.x.x.255 (Broadcast address) to any node/pod, so in practical terms there'll be only 253 available IP addresses per Subnet.

Q: Is my Pod IP Block appropriately sized for existing and future cluster workloads?

Answer: Broadcom Support cannot provide infrastructure sizing and planning guidance. However, let's walk through two example Pod IP Blocks to show what it will give you  

NOTE:

    • The recommended size of each POD IP BLOCK is /16.

    • Typically there are 7 default Namespaces for kubernetes cluster.  It can be important to take those into account during your own planning.

 

Example1:

Pod IP Block: 10.218.0.0/16 which carves out Subnets with /24 for each kubernetes Namespace (across all clusters)

Total Namespaces and Pods across all Clusters using the default /24 subnet:

      • Maximum Pods per Namespace = 253
      • Maximum Subnets (Total number of Namespaces across ALL clusters): 256
      • Total Hosts (number of Pods across all clusters): (253 * 256) = 64768

Note:  NCP/NSX won't assign three of the IP addresses to any node/pod.

For example, in a /24 Subnet there're 256 theoretical available IP addresses. NCP/NSX won't assign x.x.x.0 (Network address), x.x.x.1 (Gateway address), and x.x.x.255 (Broadcast address) to any node/pod, so in practical terms there'll be only 253 available IP addresses per Subnet.

IMPORTANT:  If the number of pods in a Namespace goes beyond 253, then an additional Subnet will be carved out of the POD IP BLOCK range and used by the same Namespace.  Thus, only 255 subnets would remain.  The number of potential Namespaces would be lowered to 255 (from 256 max subnets)

 

Example2:

Pod IP Block: 10.218.128.0/18 which carves out Subnets with /24 for each kubernetes Namespace (across all clusters)

Total Namespaces and Pods across all Clusters using the default /24 subnet:

      • Maximum Pods per Namespace = 253
      • Maximum Subnets (Total number of Namespaces across ALL clusters): 64
      • Total Hosts (number of Pods across all clusters): (253 * 64) = 16192

Note:  NCP/NSX won't assign three of the IP addresses to any node/pod.

For example, in a /24 Subnet there're 256 theoretical available IP addresses. NCP/NSX won't assign x.x.x.0 (Network address), x.x.x.1 (Gateway address), and x.x.x.255 (Broadcast address) to any node/pod, so in practical terms there'll be only 253 available IP addresses per Subnet.

IMPORTANT:  If the number of pods in a Namespace goes beyond 253, then an additional Subnet will be carved out of the POD IP BLOCK range and used by the same Namespace.  Thus, only 63 subnets would remain.  The number of potential Namespaces would be lowered to 63 (from 64 max subnets)

 

Refer to the following guides for more details:

Network Planning for Installing Tanzu Kubernetes Grid Integrated Edition with VMware NSX

Which discusses:

Pods IP Block

 

Q: How do the node_subnet_prefix, pod_subnet_prefix parameters affect the default values of the Nodes IP Block and Pods IP Block?

NOTE:

Nodes IP Block (node_ip_block_ids)  is Updatable starting with TKGI 1.16.0

Pods IP Block (pod_ip_block_ids) is also Updatable

 

Info about node_subnet_prefix:

    • Default: "node_subnet_prefix" = 24,

    • Relationship to NODE IP BLOCKS (and the node_ip_block_ids parameter):

      • This value is also used as the Default value for node_ip_block_ids.

      • Each Kubernetes cluster deployed by Tanzu Kubernetes Grid Integrated Edition owns a /24 subnet.

      • This is referenced by the Default of node_subnet_prefix of /24

        Example:

By changing the value of node_subnet_prefix to /22

Node IP Block: 10.218.0.0/16 would carves out Node Subnets with /22 per kubernetes Cluster

        • Maximum kubernetes nodes per cluster = 1021
        • Subnets (number of TOTAL Clusters): 64
        • Total Hosts (total number of cluster Nodes): (1021 * 64) = 65344

Note:  NCP/NSX won't assign three of the IP addresses to any node/pod.

For example, in a /22 Subnet there're 1024 theoretical available IP addresses. NCP/NSX won't assign x.x.x.0 (Network address), x.x.x.1 (Gateway address), and x.x.x.255 (Broadcast address) to any node/pod, so in practical terms there'll be only 1021 available IP addresses per Subnet.

Refer to the Reference on Node Subnet Prefix

 

Info about pod_subnet_prefix:

    • Default: "pod_subnet_prefix" = 24,

    • Relationship to POD IP BLOCKS (and the pod_ip_blocks_ids parameter):

      • This value is also used as the Default value for pod_ip_block_ids.

      • Each time a Kubernetes namespace is created, a subnet from the pods IP block is allocated. The default size of the subnet carved from this block for such purposes is /24 (default pod_subnet_prefix)

Example:

By changing the value of pod_subnet_prefix to /22

Node IP Block: 10.218.0.0/16 would carves out Namespace Subnets with /22 to be used across all TKGI Clusters

        • Maximum kubernetes Pods per Namespace = 1021
        • Maximum Subnets (Total number of Namespaces across ALL clusters): 64
        • Total Hosts (total number of Pods across all Clusters): (1021 * 64) = 65344

Note:  NCP/NSX won't assign three of the IP addresses to any node/pod.

For example, in a /22 Subnet there're 1024 theoretical available IP addresses. NCP/NSX won't assign x.x.x.0 (Network address), x.x.x.1 (Gateway address), and x.x.x.255 (Broadcast address) to any node/pod, so in practical terms there'll be only 1021 available IP addresses per Subnet.

Refer to the Reference on Pod Subnet Prefix

 

Q: Is my Floating IP Pool sufficient for the Kubernetes Services and cluster Load Balancers in my environment?

Answer: Floating IPs (FIPs) in TKGI when NSX Data Center is used:

    • By the Kubernetes Load Balancer services of all clusters

    • By the Kubernetes Master nodes IP(s) of all clusters

    • As SNAT IP addresses, whenever any Namespace is created (when in NAT mode) in any cluster

Answer: You can always add more Floating IP Pools if your default IP Block is exhausted or you are reaching its limit through a Network Profile

 

 

Q: Can I expand the number of Floating IPs in my environment?

Answer: Yes

 

 

Q: How do I expand available Floating IPs?

Answer:

During TKGI deployment, a single Floating IP Pool ID is populated by the TKGI Operator into the OpsManager -> TKGI tile -> Networking tab

That Floating IP Pool ID becomes the default Floating Pool ID used by all Kubernetes clusters

When deployed NSX Data Center, an operator can create more Floating IP Pool IDs and expand through a Network Profile as needed 

Steps:

    • If you run into FIP IP Exhaustion or want to expand, refer to the KB:

Add new Floating IPs (FIPs) to TKGi

    • If the plan is to update an existing Network Profile, it is important to understand:

Update-Cluster Network Profile Validation Rules

 

NOTE:

fip_pool_ids is an array used within a Network Profile.  It is Updatable. You can create additional FIP IDs to expand your Floating IP pool through a Network Profile

Not all Network Profile parameters are Updatable.  Refer to Network Profile Parameters

  •  

Q: Are there any adjustments or recommendations to improve this setup based on TKGI and NSX-T best practices?

Answer:

Broadcom Support is not able to give architectural planning guidance specific to customer business plans.  If you need architectural and related planning, customer account teams will help you with a PSO engagement to assist. 

Broadcom support recommends the following planning guides and topics: 

Network Planning for Installing Tanzu Kubernetes Grid Integrated Edition with VMware NSX

 

NSX Deployment Topologies for Tanzu Kubernetes Grid Integrated Edition

 

Considerations for Using the VMware NSX Policy API with TKGI

 

Network Profile Parameters

-> Top-Level Parameters

-> cni_configurations Parameters

 

Using Network Profiles (NSX Only)

 

Creating and Managing Network Profiles (NSX Only)

-> Network Profile Example

 

Customizing Pod Networks

Powered by Wolken Software