vCenter reports frequent failures of the “Delete a virtual storage object” task.
search cancel

vCenter reports frequent failures of the “Delete a virtual storage object” task.

book

Article ID: 412224

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

 

When vCenter is integrated with Cloud Native Storage (CNS), repeated and failing "Delete a Virtual Storage Object" tasks may occur in the vCenter UI. These tasks fail because the vStorageObject , which represents a persistent volume provisioned by CNS, cannot be deleted as it is still attached as a virtual disk to a Virtual Machine (VM).

 

The vsanvcmgmtd.log (/var/log/vmware/vsan-health/vsanvcmgmtd.log) contains entries indicating the volume is still attached to a VM and deletion is blocked by a ResourceInUse fault:

[YYYY-MM-DDTHH:MM:SS] info vsanvcmgmtd[97570] [vSAN@6876 sub=FcdService opID=d0c4ca2b] Volume 1a2b####-####-####-####-########3c4d is attached to vm vm-123#####
...
[YYYY-MM-DDTHH:MM:SS] error vsanvcmgmtd[12485] [vSAN@6876 sub=Workflow opId=d0d1f6ed] Workflow current action has fault (vim.fault.ResourceInUse) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> type = <unset>,
--> name = "volume"
--> msg = ""

To identify the VM associated with the vm-ID mentioned in the vsanvcmgmtd.log, check the vpxd.log (/var/log/vmware/vpxd/vpxd.log) for entries referencing that vm-ID:

[YYYY-MM-DDTHH:MM:SS] info vpxd[3955075] [Originator@6876 sub=OMM opID=HB-host-1098034@646040-557f29c6-WorkQueue-76c56e59] Downloading VMX config for VM [vim.VirtualMachine:vm-123####,VM_server] on host [vim.HostSystem:host-1098034,esx#####.###.####]

In this example, vm-123#### corresponds to the VM named VM_server running on host esx#####.###.####.

Cause

The underlying cause of this issue is that a vStorageObject, which was intended to be deleted  (by a CNS request for a container volume cleanup), is currently attached as a virtual disk to a running or registered VM. vCenter Server's policy dictates that it cannot delete a virtual disk (and thus its backing vStorageObject) if it is actively in use by or referenced in the configuration of a VM. This attachment prevents the deletion operation, resulting in a vim.fault.ResourceInUse error and repeated attempts by CNS to delete the volume.

Resolution

To resolve this issue, identify the VM holding the attachment to the vStorageObject and detach the virtual disk.

        1. Identify the VM and Disk Causing the Issue:

  • From the vsanvcmgmtd.log, note the volumeId (e.g., 1f9e8b3e-123f-42bf-b7b1-c347a8b14b27) and the vm-ID (e.g., vm-123#####).
  • Search vpxd.log for the vm-ID (e.g., vm-123#####) to determine the exact name of the Virtual Machine (e.g. VM_server, ) and the host it is running on. [vim.HostSystem:host-1098034,esx#####.###.####]


    2.Verify the Disk Attachment via SSH to the ESXi Host (Recommended for Confirmation)

  • Take SSH to the ESXi host where the problematic VM resides (e.g., esx#####.###.####).
  • Navigate to the VM's directory on the datastore. Example: cd /vmfs/volumes/datastore_name/VM_server/
  • For each VMDK that might be the problematic one, examine its descriptor file using cat: 

  • cat VM_server_5.vmdk could show:
    # Disk DescriptorFile
    version=5
    encoding="UTF-8"
    CID=eeeeeeee
    parentCID=ffffffff
    createType="vmfs"

    # Extent description
    RW 209715200 VMFS "VM_server-flat.vmdk"

    # Change Tracking File
    changeTrackPath="VM_server-ctk.vmdk"

    # The Disk Data Base
    #DDB
    ddb.adapterType = "lsilogic"
    ddb.deletable = "true"
    ddb.fcd.controlFlags = "1"
    ddb.fcd.createTime = "############"
    ddb.fcd.name = "pvc-#########-####-####-####-#########"
    ddb.fcd.uuid = "1a 2b ## ## ## ## ## ##-## ## ## ## ## ## 3c 4d"  <-- Match this UUID with your Volume ID
    ddb.geometry.cylinders = "13054"
    ddb.geometry.heads = "255"
    ddb.geometry.sectors = "63"
    ddb.longContentID = "###################"
    ddb.sidecars = "fcdmdsidecar, VMname.vmfd"
    ddb.thinProvisioned = "1"
    ddb.uuid = "## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##"
    ddb.virtualHWVersion = "14"

  • The ddb.fcd.uuid in the .vmdk descriptor file will directly correspond to the volumeId seen in the logs

    3.Detach the Disk from the VM:

    Note:- Before detaching any disk, it is highly recommended to power off the VM. Detaching a disk from a running VM can lead to data loss or guest OS instability. If the disk is a data disk, unmount it gracefully within the guest OS first.

  • In the vSphere Client, with the VM powered off right-click on the VM and select Edit Settings.

  • Locate the identified hard disk (e.g., "Hard Disk 5").

  • Click the "X" icon next to that specific hard disk to remove it. (Do not check the delete from disk option)

  • Click OK to apply the changes.

    4. Allow CNS to Complete Deletion:

  • Once the disk is successfully detached from the VM, the cns-volume-manager should be able to re-attempt and successfully complete the "Delete a Virtual Storage Object" task.

  • Monitor the vCenter Task & Events pane for the task to complete successfully.

Additional Information

Powered by Wolken Software