FailedAttachVolume Error in OCP Clusters Due to error `"The resource 'volume' is in use."`
search cancel

FailedAttachVolume Error in OCP Clusters Due to error `"The resource 'volume' is in use."`

book

Article ID: 454421

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • In an OpenShift Container Platform (OCP) environment, pods remain in a stuck state due to FailedAttachVolume errors.

  • Describing the affected pods reveals the following warning:

    Warning  FailedAttachVolume  <TIMESTAMP> (x76 over 10m)    attachdetach-controller AttachVolume.Attach failed for volume "pvc-25eeec64-####################" : rpc error: code = Internal desc = 
failed to attach disk: "40b10f9e-####################" with node: "######" err 
failed to attach cns volume: "40b10f9e-####################" to node vm: "VirtualMachine:vm-##### ...
LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "######"

  • Describing the VolumeAttachment associated with the volume displays a similar error regarding the node VM:

    Message: rpc error: code = Internal desc = failed to attach disk: "74f74ec6-####################" with node: "######" 
err failed to attach cns volume: "74f74ec6-####################" to node vm: "VirtualMachine:vm-##### [VirtualCenterHost: ######, UUID: 
...  LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "######"

  • The vsan-health / vsanvcmgmtd logs reflect the async detach task failing due to existing VM snapshots:

    YYYY-MM-DDTHH:MM:SS error vsanvcmgmtd[##] [vSAN@6876 sub=Workflow opId=######] Async task vim.Task:task-###### finished with fault (vim.fault.InvalidDeviceSpec) {
    faultCause = (vmodl.MethodFault) null,
    faultMessage = (vmodl.LocalizableMessage) [
       (vmodl.LocalizableMessage) {
          key = "com.vmware.vim.vm.error.removeDiskAfterVMSnapshot",
          arg = <unset>,
          message = "Cannot remove virtual disk from the virtual machine because it or one of its parent disks is part of a snapshot of the virtual machine."
       }
    ],
    property = "virtualDeviceSpec.operation",
    deviceIndex = 0
    msg = "Invalid configuration for device '0'."
 }

Environment

VMware vCenter Server
Cloud Native Storage

Cause

This issue occurs because the DetachVolume operation that must take place prior to AttachVolume fails.
Virtual machine level snapshots present on the OCP node VMs prevent the volume from detaching, causing subsequent attach tasks to detect that the volume resource is still in use.

Resolution

  1. Identify the target OCP node VMs from which the volume detachment is failing.
  2. Log into the vSphere Client and navigate to the identified virtual machines.
  3. Manually consolidate and delete the VM-level snapshots present on these VMs. Refer documentation "Deleting Snapshots " .
  4. Verify that the Container Storage Interface (CSI) can successfully complete the detach and attach operations for the volumes.
  5. Monitor the affected pods to ensure they transition out of the FailedAttachVolume state.

Additional Information

Powered by Wolken Software