Supervisor Upgrade Stalled Due to Workload Network IP Pool Exhaustion
search cancel

Supervisor Upgrade Stalled Due to Workload Network IP Pool Exhaustion

book

Article ID: 451822

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

The Supervisor cluster upgrade fails to proceed, resulting in stalled operations. The wcpsvc.log reports failure to configure the network interface with the following error:

Failed to realize NetworkInterface <vm_id>: server closed the connection while watching NetworkInterface <vm_id>: <nil> NetworkInterface <vm_id> not realized yet. Continue to watch

This issue typically manifests during the Supervisor cluster upgrade process when new control plane nodes cannot be initialized due to a lack of available IP addresses in the assigned workload network pool.

Environment

VMware vSphere Kubernetes Service

Cause

The root cause is IP address exhaustion within the workload network pool. The Supervisor upgrade process requires a specific number of available IP addresses to initialize the new Supervisor Control Plane nodes. If the available IP addresses in the workload network pool are already fully allocated (exhausted) by other services or virtual machines, the NetworkInterface fails to realize, causing the upgrade to stall.

Resolution

To resolve this issue, you must free up IP addresses in the workload network pool.

Prerequisites:

  • Ensure administrative access to the vCenter Server and the associated guest clusters.
  • Identify the workload network pool IP range currently in use.

Steps:

  1. Identify guest clusters or VMs that are utilizing the IP addresses in the target workload network pool.
  2. Scale down the worker nodes on an associated guest cluster to release the reserved IP addresses.
  3. Monitor the workload network pool to verify that the IP addresses are released back into the available pool.
  4. Once sufficient IP addresses are available, the Supervisor upgrade operation should automatically resume.

Workaround

If scaling down guest clusters is not immediately feasible, consider adding a workload network subnet or CIDR block if the environment allows for additional IP capacity, though this may require deeper network reconfiguration.

 

Additional Information

  • Review the wcpsvc.log file on the vCenter Server for specific NetworkInterface realization errors.
  • Review the NetworkInterface resources:
    kubectl get networkinterfaces.netoperator.vmware.com -n <VSPHERE_NAMESPACE>
  • Review the vmware-system-netop deployment/vmware-system-netop-controller-manager logs
    kubectl logs -n vmware-system-netop deployment/vmware-system-netop-controller-manager -c manager
    kubectl describe pod -n vmware-system-netop deployment/vmware-system-netop-controller-manager
Powered by Wolken Software