Pods fail to attach or detach volumes, input volume is not a CNS volume

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

• When using the vSphere CSI driver in a multi cluster Tanzu Kubernetes Grid Integrated Edition (TKGI) environment, pods start failing to attach or detach volumes. 
• You see messages similar to the following in the events for the affected pods:

Warning  FailedMount  25m (x2145 over 16d)    kubelet, cfbfhbeb-a32b-48df-8d30-94562da4701f  Unable to attach or mount volumes: unmounted volumes=[myvolume], unattached volumes=[myvolume]: timed out waiting for the condition

Warning  FailedAttachVolume  19m                     attachdetach-controller                        AttachVolume.Attach failed for volume "pvc-########-ffa2-123a-####-############" : rpc error: code = Internal desc = failed to attach disk: "########-abcd-4449-####-############" with node: "########-a20a-41a0-####-############" err failed to attach cns volume: "#######-abcd-4449-####-############" to node vm: "VirtualMachine:vm-172 [VirtualCenterHost: ###.###.##.##, UUID: 4237a733-67c3-8130-702c-63f9383289ba, Datacenter: Datacenter [Datacenter: Datacenter:datacenter1, VirtualCenterHost: ###.###.##.##]]". fault: "(*types.LocalizedMethodFault)(0xc0007df960)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.CnsFault) {\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\n  Reason: (string) (len=79) \"CNS: The input volume #######-abcd-4449-####-############ is not a CNS volume.\"\n },\n LocalizedMessage: (string) (len=95) \"CnsFault error: CNS: The input volume #######-abcd-4449-####-############ is not a CNS volume.\"\n})\n". opId: "074be712"

• You see messages similar to the following in the kube-controller logs on the control plane nodes:

E0616 14:11:07.519321      11 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.vsphere.vmware.com^########-1456-bcde-####-############ podName: nodeName:}" failed. No retries permitted until 2021-06-16 14:11:08.019297747 +0000 UTC m=+3981445.422084190 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"pvc-########-ffa2-123a-####-############\" (UniqueName: \"kubernetes.io/csi/csi.vsphere.vmware.com^########-1456-bcde-####-############\") from node \"########-464f-43ba-####-############\" : rpc error: code = Aborted desc = pending"
I0616 14:11:07.519367      11 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"mynamespace", Name:"mypod", UID:"dfe86621-5731-48f6-8814-b41d5318c32f", APIVersion:"v1", ResourceVersion:"80991861", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' AttachVolume.Attach failed for volume "pvc-########-ffa2-123a-####-############" : rpc error: code = Aborted desc = pending
W0616 14:11:07.521828      11 reconciler.go:206] attacherDetacher.DetachVolume started for volume "pvc-########-ffa2-123a-####-############" (UniqueName: "kubernetes.io/csi/csi.vsphere.vmware.com^########-1456-bcde-####-############") on node "########-a20a-41a0-####-############" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching
I0616 14:11:07.521874      11 reconciler.go:275] attacherDetacher.AttachVolume started for volume "pvc-27ea3883-d20a-4dbd-82ea-f346048c988c" (UniqueName: "kubernetes.io/csi/csi.vsphere.vmware.com^12345abc-abcd-sf67-8904-abc3217865c9b") from node "########-a20a-41a0-####-############"
E0616 14:11:07.668770      11 csi_attacher.go:662] kubernetes.io/csi: detachment for VolumeAttachment for volume [########-1456-bcde-####-############] failed: rpc error: code = Internal desc = volumeID "########-1456-bcde-####-############" not found in QueryVolume
E0616 14:11:07.668869      11 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.vsphere.vmware.com^########-1456-bcde-####-############ podName: nodeName:}" failed. No retries permitted until 2021-06-16 14:11:08.168818196 +0000 UTC m=+3981445.571604639 (durationBeforeRetry 500ms). Error: "DetachVolume.Detach failed for volume \"pvc-########-ffa2-123a-####-############\" (UniqueName: \"kubernetes.io/csi/csi.vsphere.vmware.com^########-1456-bcde-####-############\") on node \"########-a20a-41a0-####-############\" : rpc error: code = Internal desc = volumeID \"########-1456-bcde-####-############\" not found in QueryVolume"

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

The vSphere CSI driver uses the cluster-id for the volume create spec. If there are multiple Kubernetes clusters in the same vSphere using the same cluster-id, each time one of the Kubernetes clusters syncs to the vCenter it will tag or un-tag the volumes in vCenter. This in turn causes the volumes not to attach or detach as they should.

Resolution

Deploy each vSphere CSI driver with a unique cluster-id.

Workaround:
For already existing clusters it is possible to change the cluster-id. One of the clusters will have to keep the original cluster-id. 

1. Choose one Kubernetes cluster to keep the original cluster-id value.
2. Modify the CSI driver deployment such that the replica count is 0 on all Kubernetes clusters except for the one you selected in Step 1.
3. Wait at least one hour to allow other volumes to de-register.
4. Change the cluster-id value in all other Kubernetes clusters except for the one you selected in Step 1.
5. Modify the CSI driver deployment such that the replica count is 1 on all Kubernetes clusters except for the one you selected in Step 1.
6. Wait at least one hour to allow volumes to get re-registered.

Additional Information

This is only applicable to scenarios where CSI is manually installed. From TKGI version 1.11 and higher, CNS can be automatically enabled and this issue is not present when CNS is automatically enabled by TKGi.. See Cloud Native Storage (CNS) on vSphere for more information.

Impact/Risks:
It can take up to two hours to change cluster-ids if multiple clusters are using the same value. During that time volumes cannot be managed, and new volumes cannot be created.