Restart of a Pod takes long time to be 'Ready' state and shows Multi-attach errors when describing the Pod

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When deleting Pods, some pods stay at 'Pending' state up to around 10mins before it becomes 'Ready' state
• There are 'Multi-attach' errors when restarting Pods

  Multi-Attach error: volume is already exclusively attached and can't be attached to another node

Environment

VMware vSphere Kubernetes Service

Cause

There was a newly found code issue where 'UPDATE' method is used when 'PATCH' method should be used instead while detaching/attaching volumes to a node.
If 'UPDATE' method is used it'll lock the entire node when carrying out detach/attach operation for each volumes. So it can take much time to restart a Pod and attach, mount all volumes when there are many volumes are attached to the node.

Resolution

Currently there is no workaround, but the permanent fix using 'PATCH' method is included in VKS patch 1.33.6  By using 'PATCH' method, it is not necessary to lock an entire node when detaching/attaching a volume.