Getting "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: spec.clusterNetwork.services.cidrBlocks intersects with the network range of the container ip block" error during VKS cluster upgrade

Products

• VMware Tanzu Mission Control
• VMware vSphere Kubernetes Service

Issue/Introduction

• When trying to upgrade a VKS Cluster, it fails with the error below.
  
  API Error: Failed to upgrade cluster: failed to apply patch on cluster: (target=mc:<ID>, intentId=<ID>):
admission webhook "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: spec.clusterNetwork.services.cidrBlocks intersects with the network range of the container ip block in network provider's configuration (invalid argument)


• This issue can happen irrespective of the cluster being upgraded manually or via Tanzu Mission Control.
  
  
• Within the affected VKS Cluster, the pods have inherited IP Addresses from the service CIDR range/pool.

Environment

VMware vSphere Kubernetes Service
Tanzu Mission Control

Cause

The IP range for the services inside the cluster yaml is either incorrect or it overlaps with the IP range which has been defined for the pods. As a result, Cluster API is rejecting the request to upgrade the cluster unless this anomaly is addressed. 

Resolution

Recreate the VKS cluster with the correct IP range. Manually modifying the IP range for both the pod and service CIDR is not supported on a running cluster.

Reason: Manually updating the service CIDR range within the cluster yaml isn't recommended as it entails re-initialization of the networking stack, re-allocation of service IPs and pod subnets. This has the potential to break cluster communication and workload stability. 

Additional Information

For detailed guidance on configuring and managing network settings on Supervisor and Guest Clusters, refer the following documentation: Configuring and Managing a Supervisor

Other relevant documents can be found below.

Change the Workload Network Settings on a Supervisor Configured with VDS Networking

Change Workload Network Settings on a Supervisor Configured with NSX