VKS Supervisor Cluster upgrade is stuck : "Configured Supervisor Control plane VM's Workload Network."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• On upgrading Supervisor cluster, upgrade task does not progress and is stuck with message "Configured Supervisor Control plane VM's Workload Network".
  
  
  
  
• Pods on Supervisor Control Plane nodes are Healthy/Running state.
• New Supervisor node has been deployed but no IP is allocated to new CP VM.
• In the /var/log/vmware/wcp/wcpsvc.log file, you see entries similar to
  
  YYYY-MM-DD:T:HH:MM:SS error wcp [netoperator/networkinterfaces.go:###] [opID=########-#######-####-####-####-############] server closed the connection while watching NetworkInterface vm-######: <nil>
YYYY-MM-DD:T:HH:MM:SS error wcp [netoperator/networkinterfaces.go:##] [opID=########-#######-####-####-####-############] Failed to realize NetworkInterface vm-######: server closed the connection while watching NetworkInterface vm-######: <nil>
YYYY-MM-DD:T:HH:MM:SS debug wcp [netoperator/networkinterfaces.go:###] [opID=########-#######-####-####-####-############] Added/Modified NetworkInterface resource: vm-######
YYYY-MM-DD:T:HH:MM:SS debug wcp [netoperator/networkinterfaces.go:###] [opID=########-#######-####-####-####-############] NetworkInterface vm-###### not realized yet. Continue to watch

Environment

• vSphere Kubernetes Service - vSphere 8.0

Cause

• In vCenter > Workload Management > Supervisor Namespace, a warning appears (Workload Network <IP Pool Name> are low on free IP addresses. Details 'IPPool utilization above threshold 95%'. Edit Workload Network settings to add additional IP ranges.)
  
  

• Supervisor Cluster upgrade is stuck due to IP pool exhaustion in the Workload Network

Resolution

• Review the IP pool usage under Workload Management/NSX. If usage exceeds 90%, add a new subnet to the IP pool to extend the available IP range for allocations. Once the new subnet is added, the Supervisor Cluster upgrade should complete successfully.
• For more information VMware NSX Alarms UI displays "IP Pool Usage very high"