Guest Cluster Pods Fail to Mount Volumes Due to Pod CIDR Conflict with vSAN File Services Network

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In certain VKS environments, guest cluster pods may fail when attempting to mount  volumes. This typically manifests as repeated FailedMount warnings in pod events, with errors such as:

MountVolume.SetUp failed for volume "pvc-xxxx" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
MountVolume.SetUp failed for volume "pvc-xxxx" : rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL

Additionally, CSI pod logs may display NFS mount failures with messages like:

level=error msg="mount Failed" args="-t nfs4 -o hard,sec=sys,vers=4,minorversion=1 #####:/vsanfs/##### /var/lib/kubelet/pods/#####/volumes/kubernetes.io~csi/pvc-#####/mount" cmd=mount error="exit status 32" output="mount.nfs4: Connection refused\n"

This issue prevents pods from consuming volumes and can disrupt application availability.

Environment

VMware vSphere Kubernetes Service

Cause

The problem occurs when the Pod CIDR range of the guest cluster overlaps or conflicts with the vSAN File Services VM network CIDR.

• Example: Guest cluster Pod CIDR configured as x.x.x.x/16
• vSAN File Services VM network configured as x.x.x.x/24

Because of this overlap, kube‑proxy or antrea‑agent interprets the vSAN File Service IPs as internal cluster addresses. As a result, NFS mount traffic never leaves the cluster, and worker nodes cannot reach the vSAN File Service VMs. This leads to repeated mount failures and pod startup issues.

Resolution

To resolve the issue:

1. Uninstall the affected application to ensure workloads are safely removed.
2. Delete the existing guest cluster that was provisioned with the conflicting Pod CIDR.
3. Recreate the guest cluster with a corrected Pod CIDR range (for example, x.x.x/21) with no conflicts.

This ensures sufficient Pod CIDRs for all nodes.
Prevents overlap with the vSAN File Services VM network (x.x.x/24).
Once the new cluster is provisioned, redeploy the application.

After re‑provisioning with a non‑conflicting Pod CIDR, worker nodes can properly route traffic to vSAN File Service IPs, allowing successful NFS mounts and restoring pod functionality.

Additional Information

1. Always validate Pod CIDR ranges during guest cluster creation to avoid overlap with existing infrastructure networks (vSAN, NSX, or other management networks).
2. Consider documenting network allocations centrally to prevent CIDR conflicts across clusters.