Persistent Volume mount failure: "The resource volume is in use" error causing pod stuck in ContainerCreating or Init state

Products

• VMware Tanzu Kubernetes Grid Management
• VMware vSphere Kubernetes Service

Issue/Introduction

• The new Pod remains in the “ContainerCreating” state because it failed to mount the Persistent Volume (PV).
  
  # kubectl get pods

NAME         READY   STATUS
test-pod-1   0/1     ContainerCreating
  
  
• kubectl get volumeattachment reports ATTACHED:false

# kubectl get volumeattachment

NAME      ATTACHER                 PV        NODE          ATTACHED   AGE
csi-###   csi.vsphere.vmware.com   pvc-###   test-node-1   false      21h


• kubectl describe volumeattachment reports, "The resource 'volume' is in use."


  
  # kubectl describe volumeattachment csi-###

  Attach Error:
   Message:  rpc error: code = Internal desc = failed to attach disk: "####" with node: "###" err failed to attach cns volume: "###" to node vm: "VirtualMachine:vm-#### [VirtualCenterHost: ####, UUID: ###, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-####, VirtualCenterHost: ###]]". fault: "(*types.LocalizedMethodFault)(0xc0055730c0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (*types.ResourceInUse)(0xc0018ecf80)({\n  VimFault: (types.VimFault) {\n   MethodFault: (types.MethodFault) {\n    FaultCause: (*types.LocalizedMethodFault)(<nil>),\n    FaultMessage: ([]types.LocalizableMessage) <nil>\n   }\n  },\n  Type: (string) \"\",\n  Name: (string) (len=6) \"volume\"\n }),\n LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "<opID>"

  

• kubectl describe pod command reports, "kubelet Unable to attach or mount volumes .... timed out waiting for the condition"
  

  # kubectl describe pod <pod_name> 

Warning  FailedAttachVolume  33s (x2 over 2m35s)    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-###" : rpc error: code = Internal desc = failed to attach disk: "###" with node: "###" err failed to attach cns volume: "###" to node vm: "VirtualMachine:vm-### [VirtualCenterHost: ###, UUID: ###, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-####, VirtualCenterHost: ###]]". fault: "(*types.LocalizedMethodFault)(0xc0011f0f00)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (*types.ResourceInUse)(0xc000f4cac0)({\n  VimFault: (types.VimFault) {\n   MethodFault: (types.MethodFault) {\n    FaultCause: (*types.LocalizedMethodFault)(<nil>),\n    FaultMessage: ([]types.LocalizableMessage) <nil>\n   }\n  },\n  Type: (string) \"\",\n  Name: (string) (len=6) \"volume\"\n }),\n LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "<opID>"

  Warning  FailedMount         21s (x3 over 4m53s)    kubelet  Unable to attach or mount volumes: unmounted volumes=[###], unattached volumes=[###], failed to process volumes=[]: timed out waiting for the condition

Environment

• Tanzu Kubernetes Grid Management
• vSphere Kubernetes Service
• vSphere Container Storage Plug-in

Cause

The Persistent Volume could not be mounted to the scheduled node (for example, test-node-1) because it was still attached to another worker node. Although the VolumeAttachment resource shows the PV as mounted on a node (test-node-1), in reality it remains attached elsewhere.

Resolution

Before applying the Resolution in this KB, verify if the following KB is applicable : PV attachment to a pod fails with the error: 'The resource volume is in use'.

1. Identify the Worker node VM name that has already mounted the target Persistent Volume by checking in vCenter UI:
   The "Volume Path" is used in the next step to identify the disk
   
   
   
   
2.  Manually detach the Persistent Volume from the target worker node VM.
   1. Select the target worker VM --> Edit Settings
   2. Check Disk File corresponds with the Volume Path
   3. Detach the Persistent Volume
      
      
      
      
3. Recreate the target pod to remount the Persistent Volume.
   kubectl -n <namespace> delete pod <pod_name>

Additional Information

For vSAN environment: "FcdService AttachDisk already attached" Error when attempting to attach a persistent volume to a Kubernetes node