Pods are not coming up on newly created node due to volume attachment bound to old deleted machine object

Products

• VMware Telco Cloud Automation

Issue/Introduction

When we delete a machine object due to an issue with the node, at times the Volume attachments does not get deleted and will cause an issue for other newly created pods/nodes from gaining access to the persistent volumes.

In the pod logs you will see these messages:

 

Warning        FailedAttachVolume         50m     attachdetach-controller         Multi-Attach error for volume pvc-#########-####-####-####-############" Volume is already exclusively attached to one nodWarning e and can't be attached to another
Warning        FailedMount             50m     kubelet                            Unable to attach or mount volumes: unmounted volumes=[pod name1, pod name2, pod name3], unattached volumes=[VolumeName1 VolumeName2 VolumeName3]: timed out waiting for the condition

Environment

2.x

Resolution

In order to resolve the issue we have to remove the old volumeattachments resources

1. Find the persistent volumes given in the error in the pod logs and confirm if it exists
   

   kubectl get pv -A | grep -i pvc-#########-####-####-####-############

2. Find the volume attachments filtering with old node name
   

   kubectl get volumeattachments | grep -i NodeName

3. Delete the volumeattachment
   

   kubectl delete volumeattachment csi-#######################