Persistent volumes cannot attach to a new node if previous node is deleted

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Due to race condition between detaching and deleting volume operations, CNS volumes never get detached from the nodes. This is a known issue  with the vSphere CSI driver version shipped with TKGi.

One of the scenarios in which this can occur is during an upgrade and if you have stateful workloads utilizing persistent volumes:
 

• TKGi upgrade hung due to PDB
• After resolving the PDB errors workers get reconciled and old workers get deleted
• CSI controller repeatedly tries to detach the volume from a node that is not present
• All stateful workloads are stuck in container creation or an init state, depending on which stage the volume is mounted

Repeated error messages "Failed to find VirtualMachine for node" are logged in the vSphere CSI controller logs:

kubectl logs -n kube-system vsphere-csi-controller-76d888d87c-wsml9 vsphere-csi-controller

}{{"level":"error","time":"2020-09-27T18:15:59.174108121Z","caller":"vanilla/controller.go:569","msg":"failed to find VirtualMachine for node:\"tcp-md-0-5bb7dc9f5c-mbjwl\". 
Error: node wasn't found","TraceId":"a5ab0f92-a59e-4b67-9185-a9bd020cc1fb","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).ControllerUnpublishVolume\n\t/build/pkg/csi/service/vanilla/controller.go:569\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerUnpublishVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5200\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).controllerUnpublishVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:141\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:88\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerUnpublishVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5202\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}

The application pods are stuck during the container creation or init container phase and constantly error out with "Multi-Attach error for volume and Unable to attach or mount volumes: unmounted volumes":

kubectl describe <your-failing-pod>

Events:
  Type     Reason              Age                  From                                Message
  ----     ------              ----                 ----                                -------
  Normal   Scheduled           32m                  default-scheduler                   Successfully assigned default/wordpress-6c6794cb7d-cdnsc to tcp-md-0-7f67dbbfb8-lthnt
  Warning  FailedAttachVolume  32m                  attachdetach-controller             Multi-Attach error for volume "pvc-c3c29367-658b-4548-ac7c-134fa73df4c2" Volume is already exclusively attached to one node and can't be attached to another
  Warning  FailedMount         12m (x2 over 21m)    kubelet, tcp-md-0-7f67dbbfb8-lthnt  Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[istio-certs default-token-wdbzv wordpress-persistent-storage istio-envoy]: timed out waiting for the condition
  Warning  FailedMount         9m49s (x2 over 30m)  kubelet, tcp-md-0-7f67dbbfb8-lthnt  Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[wordpress-persistent-storage istio-envoy istio-certs default-token-wdbzv]: timed out waiting for the condition
  Warning  FailedMount         5m19s (x3 over 23m)  kubelet, tcp-md-0-7f67dbbfb8-lthnt  Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[istio-envoy istio-certs default-token-wdbzv wordpress-persistent-storage]: timed out waiting for the condition
  Warning  FailedMount         44s (x7 over 27m)    kubelet, tcp-md-0-7f67dbbfb8-lthnt  Unable to attach or mount volumes: unmounted volumes=[wordpress-persistent-storage], unattached volumes=[default-token-wdbzv wordpress-persistent-storage istio-envoy istio-certs]: timed out waiting for the condition

Environment

Product Version: 1.0
OS: PhotonOS

Resolution

There are 2 ways to resolve issue

1. Apply workaround

Check the current pod and node status:

kubectl get nodes,pods | grep -v Running
NAME                             STATUS   ROLES    AGE   VERSION
node/tcp-control-plane-4z4nh     Ready    master   9h    v1.18.8+vmware.1
node/tcp-control-plane-bd9mq     Ready    master   9h    v1.18.8+vmware.1
node/tcp-control-plane-fxq4l     Ready    master   9h    v1.18.8+vmware.1
node/tcp-md-0-7f67dbbfb8-7xhsh   Ready    <none>   9h    v1.18.8+vmware.1
node/tcp-md-0-7f67dbbfb8-ggzx8   Ready    <none>   9h    v1.18.8+vmware.1
node/tcp-md-0-7f67dbbfb8-lthnt   Ready    <none>   9h    v1.18.8+vmware.1
node/tcp-md-0-7f67dbbfb8-v64fh   Ready    <none>   9h    v1.18.8+vmware.1
 
NAME                                   READY   STATUS     RESTARTS   AGE
pod/web-0                              0/2     Init:0/1   0          9h
pod/wordpress-6c6794cb7d-cdnsc         0/2     Init:0/1   0          31m
pod/wordpress-mysql-756d555798-gtvvp   0/2     Init:0/1   0          9h

Query the existing volumeattachments and compare the node names to the nodes from the output above:

kubectl get volumeattachments.storage.k8s.io
NAME                                                                   ATTACHER                 PV                                         NODE                        ATTACHED   AGE
csi-11a0b1c040fd7179707f982ea0cc3856bafe5ffef47217d7c059fd91fb1fa9d1   csi.vsphere.vmware.com   pvc-bcb3f6a3-8d13-4187-ac5c-4ff4c0a4cbbb   tcp-md-0-5bb7dc9f5c-mbjwl   true       12h
csi-356f081d7d4592f6ee2e6bd9d99c1ce74f8878d9b09db654582c12a508088b95   csi.vsphere.vmware.com   pvc-9544f100-d12e-4896-8222-f80664d51906   tcp-md-0-5bb7dc9f5c-fdvgj   true       12h
csi-81e0bcfd89e8c3d0c802440fd728ef57841199f65f0d70176eaac9988f681762   csi.vsphere.vmware.com   pvc-c3c29367-658b-4548-ac7c-134fa73df4c2   tcp-md-0-5bb7dc9f5c-rbwfk   true       13h
csi-a939fdd945e9b1eebf118ed4af611212164e58af9b34a1c1fe6ff1846bdd4c92   csi.vsphere.vmware.com   pvc-ff9346f6-aaba-43c7-9dab-fac6e2aedeef   tcp-md-0-5bb7dc9f5c-fdvgj   true       12h
csi-c05c631ca9c2d427d7ce64843052a3a9558765f83168b739e83f13ef94d506da   csi.vsphere.vmware.com   pvc-5cbffda0-ac29-4924-8e65-ac297895974a   tcp-md-0-5bb7dc9f5c-fdvgj   true       13h
csi-fdce83b0bffb218ebde46042c40c58bae74df14e7677fd0ddb4c9d3a2424236c   csi.vsphere.vmware.com   pvc-35f6c891-5d4d-4685-b2cb-c71505f063bc   tcp-md-0-5bb7dc9f5c-mbjwl   true       12h

On comparing the above two outputs, it is clear that there are certain volumeattachment objects which refer to nodes that are no longer part of the cluster. These attachments need to be deleted to workaround the multi attach errors. Before deleting the volume attachments, please make sure attachment of only those nodes are deleted that are not part of the kubectl get nodes command's output.

To delete the attachments, remove the finalizer from all the volumeattachments that belonged to older nodes. Example:

kubectl patch volumeattachments.storage.k8s.io csi-11a0b1c040fd7179707f982ea0cc3856bafe5ffef47217d7c059fd91fb1fa9d1 -p '{"metadata":{"finalizers":[]}}' --type=merge 

The new volume attachments should soon be created and new nodes will be able to mount the persistent volumes.