TKC cluster stuck in creating state with error "exhausted all ip addresses in requested ip pools"

Products

• Tanzu Kubernetes Runtime
• VMware vSphere 7.0 with Tanzu

Issue/Introduction

Unable to deploy/upgrade the TKC cluster as it is stuck in a pending state.

The OVF template is getting deployed but stuck in the network interface stage. 

Below is the error message noticed in the netops controller pod for haproxy load balancer, ncp pod for NSXT LB, and ako pod for NSXT-AVI LB .

EDDMM HH:MM:SS.XXXXXX 1 network interface_controller.go:250] controllers/NetworkInterface/XXXX/control-plane-VM "msg"="error reconciling NetworkINterfWin dows"-"exhausted all ip addresses in requested ip pools"

EDDMM HH:MM:SS.XXXXXX 1 controller.go:317] controller/network interface "msg"="Reconciler error" "error"="exhausted all ip addresses in requested ip pools" "name"="control-plane-VM alecatcontreduBlane -XXXX" "namespace"="XXXX" "reconciler group"="netoperator.vmware.com" "reconciler kind"="NetworkInterface"

Environment

Vmware vSphere with Tanzu 

Cause

The log entries mean that the NetworkInterface controller is failing to reconcile due to the exhaustion of IP addresses in the IP pools. No more IPs are available.

Resolution

Pods are running in the supervisor VM, like apiserver, and the VMservice pods use the workload network and IP is shared between them.

Expand the existing IP pool or add the new subnet. Below is the document to add it workload network to the supervisor cluster.

Add workload network to a supervisor cluster vCenter 7.0 

Add workload network to a supervisor cluster vCenter 8.0 

Additional Information

Refer to the doc for more information about the network requirements to configure the workload networks.