Pod mount ReadWriteMany PVC fails with An operation with the given Volume ID already exists

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• vSAN File Service has been enabled to provide ReadWriteMany PVC for TKG cluster.
• Pod in TKG cluster fails to mount ReadWriteMany PVC and describe the pod with below similar messages:
  
  Warning  FailedMount  47m (x6 over 48m) kubelet MountVolume.SetUp failed for volume "pvc-<id>" : rpc error: code = Aborted desc = NodePublishVolume failed: An operation with the given Volume ID <volume-id> already exists
  
  
• In the work node where the Pod is running check the CSI Node pod log with below command: 
  
  kubectl logs -n vmware-system-csi vsphere-csi-node-##### -c vsphere-csi-node

  
  • Task 'NodePublishVolume' runs two times.
  • The second time of the task fails with above error that: An operation with the given Volume ID <id> already exists
  • That is because the first time running of task has not finished yet. 
  • After 3 minutes the first time running of the task failed too with below messages:
    
    2025-04-02T05:33:26.601272573Z stderr F time="2025-04-02T05:33:26Z" level=error msg="mount Failed" args="-t nfs4 -o hard,sec=sys,vers=4,minorversion=1 <vsan-file-server-fqdn>:/vsanfs/<nfs-uuid> /var/lib/kubelet/pods/<pod-id>/volumes/kubernetes.io~csi/pvc-<id>/mount" cmd=mount error="exit status 32" output="mount.nfs4: Connection timed out\n"

Environment

vSphere with Tanzu

Cause

Port 2049 TCP and UDP is blocked by firewall to access vSAN file service for NFS. The port must be open for the workload network of TKG cluster to access vSAN file service NFS. NFSv3 can use both TCP and UDP ports but NFSv4.1 uses only TCP.

Resolution

To resolve the issue, open the port 2049 for TCP and UDP of vSAN file server to allow accessing from TKG cluster work node. 

More information about vSAN file service networking, see Networking Considerations for vSAN File Service