When we delete a machine object due to an issue with the node, at times the Volume attachments does not get deleted and will cause an issue for other newly created pods/nodes from gaining access to the persistent volumes.
In the pod logs you will see these messages:
Warning FailedAttachVolume 50m attachdetach-controller Multi-Attach error for volume pvc-#########-####-####-####-############" Volume is already exclusively attached to one nodWarning e and can't be attached to another
Warning FailedMount 50m kubelet Unable to attach or mount volumes: unmounted volumes=[pod name1, pod name2, pod name3], unattached volumes=[VolumeName1 VolumeName2 VolumeName3]: timed out waiting for the condition
2.x
In order to resolve the issue we have to remove the old volumeattachments resources
kubectl get pv -A | grep -i pvc-#########-####-####-####-############
kubectl get volumeattachments | grep -i NodeName
kubectl delete volumeattachment csi-#######################