RWM Volume become inaccessible if Node Daemonset Pod which helped mount the volume is restarted.

Products

• VMware

Issue/Introduction

When customer is upgrading CSI driver to 2.3 release, Node Daemonset pods will be recreated with new images and if customers are using vSAN file share volumes on the cluster, those volumes will become inaccessible.
We want customer to be aware about this issue and want them to follow upgrade instructions for 2.3 release.

Symptoms:
Application Pod using this volume will not be able to write/read data from the inaccessible volume. When Application Pod needs to be restarted on another node, It will be stuck in the terminating state, as unmount operation by Node Daemonset will fail to unmount the volume.

Cause

This issue occurs because Node Daemonset Pod has mounted the volume, when this Pod is restarted, new Pod is coming up with different IP address. NFS Client hand over is not happening in this case, and thus volume goes in the inaccessible states.

Resolution

This issue is resolved in vSphere CSI Driver v2.3.0.

All previous versions are impacted

Workaround:
Please follow the steps detailed in the link

https://vsphere-csi-driver.sigs.k8s.io/driver-deployment/upgrade.html#if-you-have-rwm-volumes-backed-by-vsan-file-service-deployed-using-vsphere-csi-driver-please-refer-to-the-following-steps-before-upgrading-vsphere-csi-driver