vSAN File Service PV Mount Fails with DeadlineExceeded on Pods

Products

• VMware vSphere Kubernetes Service
• VMware vSAN

Issue/Introduction

• When a Kubernetes pod attempts to mount a ReadWriteMany (RWX) Persistent Volume backed by vSAN File Services, the pod remains stuck in ContainerCreating or Unknown states.
• Executing kubectl describe pod reveals the following sequence of errors in the Events section:
• Initial Mount Timeout: Warning FailedMount kubelet MountVolume.SetUp failed for volume "pvc-<SECRET>" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
• Subsequent CSI Driver Blocks: Warning FailedMount kubelet MountVolume.SetUp failed for volume "pvc-<SECRET>" : rpc error: code = Aborted desc = NodePublishVolume failed: An operation with the given Volume ID <SECRET> already exists.

Environment

• VMware vSphere Kubernetes Service
• VMware vSAN File Services

Cause

• The Kubernetes worker node cannot resolve the backend vSAN file server hostname via DNS after receiving an NFSv4 Referral (NFS4ERR_MOVED) from the primary vSAN IP. This causes the Linux kernel to abort the connection with an EHOSTUNREACH (No route to host) error. Because the connection drops within the kernel's routing/DNS layer, the Kubernetes kubelet hangs waiting for a response until it times out.

Resolution

1. SSH into the problematic Kubernetes worker node.

2. Verify Layer 4 network connectivity to ensure firewalls are not blocking the connection to the primary vSAN File Service IP: curl -v telnet://<IP>:2049 curl -v telnet://<IP>:111

3. Attempt a manual OS-level mount to bypass Kubernetes and confirm the No route to host failure: mkdir -p /mnt/nfstest mount -t nfs -v -o vers=4 <IP>:/vsanfs/<SECRET> /mnt/nfstest

4. Enable NFS Kernel Debugging to reveal the NFSv4 referral failure (-113 translates to EHOSTUNREACH) and then disable debugging:
   echo 32767 > /proc/sys/sunrpc/rpc_debug && echo 32767 > /proc/sys/sunrpc/nfs_debug && mount -t nfs -v -o vers=4 <IP>:/vsanfs/<SECRET> /mnt/nfstest ; dmesg | tail -n 200 | grep -iE 'referral|location|fs_location|destination|migration|113' ; echo 0 > /proc/sys/sunrpc/rpc_debug && echo 0 > /proc/sys/sunrpc/nfs_debug

5. Log into vCenter.

6. Navigate to the vSAN Cluster > Monitor > vSAN > Skyline Health.

7. Expand File Server Connectivity. Verify the presence of health check failures marked in red with the description: "Both DNS forward and reverse lookup are not working or incorrect" alongside specific vSAN backend hostnames (e.g., <HOSTNAME>).

8. Create the missing DNS records on the corporate DNS infrastructure for the failing hostnames:

   1. Forward Record (A): Map the vSAN Node Hostname to its IP Address.

   2. Reverse Record (PTR): Map the vSAN Node IP Address back to its Hostname.

Additional Information

• If Layer 4 connectivity tests time out on ports 2049 or 111, this indicates a standard firewall issue. Refer to Knowledge Base articles 432736 and 400625.