K8s pods fail to mount vSAN File Service PV with error: "NodePublishVolume failed: An operation with the given Volume ID already exists"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When a K8s pod attempts to mount a Persistent Volume on vSAN File Service, it fails with the following error:

# kubectl describe pod <POD_NAME>
Warning  FailedMount  47m (x6 over 48m) kubelet MountVolume.SetUp failed for volume "pvc-<id>" : rpc error: code = Aborted desc = NodePublishVolume failed: An operation with the given Volume ID <volume-id> already exists

Environment

• vSphere Kubernetes Service
• vSAN File Service

Cause

A malfunction in the vSAN File Service component.

Resolution

First, SSH into the worker node and verify connectivity to the vSAN File Service.

curl -v telnet://<VSAN_FILE_SERVICE_IP>:2049 # Conenected or not

If it is unreachable, refer to the following KB article:

KB - Pod mount ReadWriteMany PVC fails with An operation with the given Volume ID already exists

If the issue persists, open a support case with the vSAN Support team and reference this KB.

Additional Information

Japanese KB: https://knowledge.broadcom.com/external/article/437025