Attaching vSAN FileShare (RWX) volumes to vSphere Pods fails

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When attempting to deploy a vSphere Pod workload in the Supervisor Namespace using a PersistentVolumeClaim (PVC) configured with the ReadWriteMany (RWX) access mode, the volume attachment fails.
• The deployment works correctly when created from a VKS (or TKG) Clusters
• Status of vSphere pod reports ProviderFailed
  • kubectl get pods-n example_namespace

NAME                                    READY   STATUS           RESTARTS   AGE
pod/nginx-deployment-5d8###4f66-kqgzg   0/1     Pending          0          8m25s
pod/nginx-deployment-5d8###4f66-vdrq7   0/1     ProviderFailed   0          10s
pod/nginx-deployment-5d8###4f66-l2m59   0/1     ProviderFailed   0          18m

• The Kubernetes events for the pod show a Warning with FailedAttachVolume:
  • kubectl get events -A | grep -i nginx-deployment-5d8###f66-vdrq7

6s          Warning   FailedAttachVolume             pod/nginx-deployment-5d8###f66-vdrq7                         AttachVolume.Attach failed for volume "pvc-2b111af0-28d9-####-a357-eabf5d31bdca" : rpc error: code = Internal desc = failed to attach volume with volumeID: file:b6f7b5a6-6fbe-4953-####-8d6c88867191. Error: ServerFaultCode: Permission to perform this operation was denied.

• The corresponding container File Share may be successfully created, but its "Net access control" is set to "no one can access"
  • vSAN logs on vCenter server at /var/log/vmware/vsanvcmgmtd.log: 

[timestamp] info vsanvcmgmtd[21786] [vSAN@6876 sub=FileService opID=89###d2] Finish creating fileshare id=file:b6f7b5a6-6fbe-4953-####-8d6c88867191
[timestamp] info vsanvcmgmtd[21786] [vSAN@6876 sub=VsanFileService opID=89###d2] Successfully fileshare created: fileId : file:b6f7b5a6-6fbe-4953-####-8d6c88867191 "file:b6f7b5a6-6fbe-4953-####-8d6c88867191"
[timestamp] error vsanvcmgmtd[341452] [vSAN@6876 sub=FcdService opID=89###7db] Failed to find vol file:b6f7b5a6-6fbe-4953-####-8d6c88867191 from volume[timestamp] infoCache

Cause

File volumes are supported only for workloads in the VKS cluster. They are not supported for workloads, such as vSphere Pods and VM Service VMs, in the Supervisor namespace.

Resolution

This behavior is by design, as a workaround follow either of the below options: 

Option 1:

Reconfigure the PVC accessModes to a supported mode such as ReadWriteOnce (RWO).

Option 2:

Deploy the workload to a VKS (or TKG) Cluster. The ReadWriteMany (RWX) access mode using vSAN File Service is fully supported for pods running within a TKG Guest Cluster.

Note: For the official product documentation refer to: Using vSAN File Service to Create ReadWriteMany Volumes in Supervisor Environment