Pod Stuck in "ContainerCreating" or "Init" state due to Persistent Volume Mount Failure: “The resource volume is in use.”
Article ID: 411087
Updated On:
Products
Issue/Introduction
The new Pod remains in the “ContainerCreating” state because it failed to mount the Persistent Volume (PV).
# kubectl get pods
NAME READY STATUS
test-pod-1 0/1 ContainerCreating
kubectl get volumeattachment reported ATTACHED:false
# kubectl get volumeattachment
NAME ATTACHER PV NODE ATTACHED AGE
csi-### csi.vsphere.vmware.com pvc-### test-node-1 false 21h
kubectl describe volumeattachment reported"The resource 'volume' is in use.".
# kubectl describe volumeattachment csi-###
Attach Error:
Message: rpc error: code = Internal desc = failed to attach disk: "####" with node: "###" err failed to attach cns volume: "###" to node vm: "VirtualMachine:vm-#### [VirtualCenterHost: ####, UUID: ###, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-####, VirtualCenterHost: ###]]". fault: "(*types.LocalizedMethodFault)(0xc0055730c0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (*types.ResourceInUse)(0xc0018ecf80)({\n VimFault: (types.VimFault) {\n MethodFault: (types.MethodFault) {\n FaultCause: (*types.LocalizedMethodFault)(<nil>),\n FaultMessage: ([]types.LocalizableMessage) <nil>\n }\n },\n Type: (string) \"\",\n Name: (string) (len=6) \"volume\"\n }),\n LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "2ea0f4e7"
kubectl describe pod command reported "kubelet Unable to attach or mount volumes .... timed out waiting for the condition"
# kubectl describe pod <pod_name>
Warning FailedAttachVolume 33s (x2 over 2m35s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-###" : rpc error: code = Internal desc = failed to attach disk: "###" with node: "###" err failed to attach cns volume: "###" to node vm: "VirtualMachine:vm-### [VirtualCenterHost: ###, UUID: ###, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-####, VirtualCenterHost: ###]]". fault: "(*types.LocalizedMethodFault)(0xc0011f0f00)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (*types.ResourceInUse)(0xc000f4cac0)({\n VimFault: (types.VimFault) {\n MethodFault: (types.MethodFault) {\n FaultCause: (*types.LocalizedMethodFault)(<nil>),\n FaultMessage: ([]types.LocalizableMessage) <nil>\n }\n },\n Type: (string) \"\",\n Name: (string) (len=6) \"volume\"\n }),\n LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "2ea0f022"
Warning FailedMount 21s (x3 over 4m53s) kubelet Unable to attach or mount volumes: unmounted volumes=[###], unattached volumes=[###], failed to process volumes=[]:timed out waiting for the condition
Environment
- Tanzu Kubernetes Grid Management
- vSphere Kubernetes Service
- vSphere Container Storage Plug-in
Cause
The Persistent Volume could not be mounted to the scheduled node (for example, test-node-1) because it was still attached to another worker node. Although the VolumeAttachment resource shows the PV as mounted on a node (test-node-1), in reality it remains attached elsewhere.
Resolution
Before applying the Resolution in this KB, please check whether any of this KB is applicable.
PV attachment to a pod fails with the error: 'The resource volume is in use'.
1. Identify the Worker node VM name that has already mounted the target Persistent Volume by checking in vCenter.
The "Volume Path" is used in the next step to identify the disk.
2. Manually detach the Persistent Volume from the target worker node VM.
- Select the target worker VM --> Edit Settings
- Check Disk File corresponds with the Volume Path
- Detach the Persistent Volume
3. Recreate the target pod to remount the Persistent Volume.
kubectl -n <namespace> delete pod <pod_name>Additional Information
- prometheus pod stuck on ContainerCreating due to PV attachment
- This is for VKS environment
- "FcdService AttachDisk already attached" Error when attempting to attach a persistent volume to a Kubernetes node
- This is for vSAN environment
Feedback
