Persistent volumes cannot attach to a new Tanzu Kubernetes Grid 1.2 node if the previous node is deleted

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• Issue impacts pods with a persistent volume attached
• If the worker node is deleted, the pod does not come up on another node
• You see the pod gets stuck at ContainerCreating state
• Only happens when using CSI providers

Environment

VMware Tanzu Kubernetes Grid 1.x

Resolution

This is a known issue affecting Tanzu Kubernetes Grid 1.2. There is currently no resolution.

Workaround:
To workaround the issue, edit the volumeattachment linked to the PV attached to the pod and remove the finalizer.

To edit the volumeattachment run,

1. kubectl edit volumeattachment csi-xxxxxxxxxxxxxxxxxx
2. Remove the line saying finalizer and the next line which says external-attacher/csi-vsphere-vmware-com
3. Save and exit

Additional Information

You can also see this issue in any vSphere Container Storage Interface (CSI) 2.0.0 implementation (vanilla Kubernetes, vSphere with Tanzu, Tanzu Kubernetes Grid Integrated Edition). The Known Issues section of vSphere Container Storage Plug-in 2.0.0 provides more details on this issue and the workaround.