Error: "ClusterNetworkProvisionFailed - Cannot realize subnet" during VKS cluster creation in VCF 9

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When attempting to provision a new VKS cluster in a VCF 9.x environment, the cluster may remain in a Provisioning state indefinitely.
• When creating a VKS cluster, the following events may appear:

InfrastructureReadyFalseWarningClusterNetworkProvisionFailedCannot realize subnet
TopologyReconciledTrue
StorageProviderReconciledFalseWarningStorageProviderNotInstalled
NetworkProviderReconciledFalseWarningContainerNetworkingNotInstalled
ControlPlaneReadyFalseWarningScalingUpScaling up control plane to 1 replicas (actual 0)
ControlPlaneInitializedFalseInfoWaitingForControlPlaneProviderInitializedWaiting for control plane provider to indicate the control plane has been initialized
CloudProviderReconciledFalseWarningCloudProviderNotInstalled
ReadyFalseWarningScalingUpScaling up control plane to 1 replicas (actual 0)
UpdatesAvailableFalseInfoAlreadyUpToDate

• Describing the cluster shows the following error

kubectl describe cluster <cluster-name> -n <namespace>

InfrastructureReady:
  NetworkReady: Cannot realize subnet
  LoadBalancerReady: Condition not yet reported

• Additional logs indicate that the namespace subnet capacity has been exhausted.

kubectl get events -n <namespace>

LAST SEEN   TYPE      REASON                 OBJECT                             MESSAGE
3m26s       Warning   IPUsageOverThreshold   namespacenetworkinfo/<namespace>   subnetCIDRUsage has reached the usage threshold 0.8

kubectl describe namespacenetworkinfo <cluster-name> -n <namespace>

Usage:
  Egress CIDR Usage:
    Allocated:  0
    Total:      0

  Ingress CIDR Usage:
    Allocated:  0
    Total:      64

  Subnet CIDR Usage:
    Allocated:  64
    Total:      64

Warning   IPUsageOverThreshold   nsx-container-ncp   subnetCIDRUsage has reached the usage threshold 0.8.

• When navigating to NSX > Segments, the corresponding segment does not show any connected ports or interfaces.

 

Environment

vSphere Kubernetes  Service

Cause

• The VKS Namespace network configuration contains a CIDR mask for the Namespace Network that is identical to the Namespace Subnet Prefix.

Example configuration:

Namespace Network: <namespace-network>/26   (Total 64 addresses)
Namespace Subnet Prefix: /26                (Requires 64 addresses per subnet)

• Since each VKS cluster requires its own dedicated subnet, using the same prefix size for both the namespace network and subnet prefix leaves no additional subnet space available for new cluster deployments.

Resolution

• To resolve this issue, Recreate the cluster and Modify the VKS namespace network configuration to provide a larger CIDR block while allocating smaller subnets per cluster.

Example configuration:
Namespace Network: <namespace-network>/24(256 addresses)
Namespace Subnet Prefix: /28(16 addresses per subnet)

• This configuration allows the /24 namespace network to be divided into multiple /28 subnets, Each VKS cluster can receive a dedicated subnet, enabling successful provisioning.