Error "Insufficient disk space on datastore" when deleting a VM disk from VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• When deleting a disk from VM which has 2 disks, placed on two different datastores, VCD is observed to initiate a relocate VM Disk task.
  • For ex: vmx and Disk1 on DS1 and Disk2 on DS2. and we attempt to delete disk2, VCD initiates a svMotion of disk2 to DS1 before deletion.
• The 'Relocate Virtual Machine' task in vCenter is triggered for the VM but it fails with the following error:
  Insufficient disk space on datastore 'DATASTORE_NAME'
• The task in VMware Cloud Director flags a similar error:

null - Task failed, vc-id={VC_ID}, task-moref={ManagedObjectReference: type = Task, value = task-xxxxx, serverGuid = null}, error={Insufficient disk space on datastore 'DATASTORE_NAME'.} - Insufficient space on datastore DATASTORE_NAME'. - Insufficient disk space on datastore 'DATASTORE_NAME''.

• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log you have the similar errors:

2023-12-11 23:40:11,814 | ERROR | task-service-activity-pool-1007 | FutureUtil | Failed to relocate VM to a Hub | requestId=REQUEST_ID,request=POST https://vcloud.example.com/api/vApp/vm-VM_ID/action/reconfigureVm,requestTime=1702338002871,remoteAddress=xx.xx.xx.xx:port,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 38.0.0-alpha
 vcd=VCD_ID,task=TASK_ID activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx)
com.vmware.ssdc.util.RelocateException: null
        at com.vmware.vcloud.val.internal.impl.RelocateVmActivity$WaitForInventoryUpdatePhase.invoke(RelocateVmActivity.java:303)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.vcloud.val.exceptions.TaskManagerException: Task failed, vc-id={VC_ID}, task-moref={ManagedObjectReference: type = Task, value = task-xxxxxx, serverGuid = null}, error={Insufficient
disk space on datastore 'DATASTORE_NAME'.}
        at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$WaitForInventoryUpdatePhase.invoke(AsynchronousTaskWaitActivity.java:279)
        ... 7 more
Caused by: com.vmware.ssdc.util.LMException: Insufficient space on datastore DATASTORE_NAME.
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:412)
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:499)
        ... 8 more
Caused by: (vim.fault.NoDiskSpace) {
   faultCause = null,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         dynamicType = null,
         dynamicProperty = null,
         key = com.vmware.vim.datastore.error.noSpaceExtendedInfo,
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               dynamicType = null,
               dynamicProperty = null,
               key = requiredSpace,
               value = 3221225472000
            },
            (vmodl.KeyAnyValue) {
               dynamicType = null,
               dynamicProperty = null,
               key = freeSpace,
               value = 1170505465856
            }
         ],
         message = Required space: 3221225472000. Available space: 1170505465856.
      }
   ],
   file = [DATASTORE_NAME] VM_NAME/VM_NAME.vmdk,
   datastore = DATASTORE_NAME
}

• On vCenter /log/var/log/vmware/vpxd.log an error similar to below:

2023-12-11T23:40:15.847Z info vpxd[09761] [Originator@6876 sub=Default opID=vcd-OPERATION_ID;activity=urn:uuid:xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx-xx] [VpxLRO] -- ERROR task-xxxxxxx -- vm-VM_MOREF -- vim.VirtualMachine.relocate: vim.fault.NoDiskSpace:
--> Result:
--> (vim.fault.NoDiskSpace) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = (vmodl.LocalizableMessage) [
--> (vmodl.LocalizableMessage) {
--> key = "com.vmware.vim.datastore.error.noSpaceExtendedInfo",
--> arg = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "requiredSpace",
--> value = 3221225472000
--> },
--> (vmodl.KeyAnyValue) {
--> key = "freeSpace",
--> value = 1170505465856
--> }
--> ],
--> message = <unset>
--> }
--> ],
--> file = "[DATASTORE_NAME] VM_NAME/VM_NAME.vmdk",
--> datastore = "DATASTORE_NAME]"
--> msg = ""
--> }

Environment

VMware Cloud Director 10.4.x
VMware Cloud Director 10.5.x

Cause

• The VM relocation is triggered because the VM is powered on, and the host selection flag is set to true:
  [type:HostSelectionRequired, value:true]
• Since the destination datastore doesn't have sufficient space, it flags the failure stating insufficient space. 

Resolution

The is a known issue impacting VMware Cloud Director 10.4.x and 10.5.x and has been resolved in 10.6.0. release. To prevent the relocation of VM disks during deletion, upgrade your VMware Cloud Director to 10.6.0 or later. 

Workaround:
There are three possible ways to workaround this issue:

1. Deleting the disk with the VM power off, to prevent relocation.
2. Removing the disk directly from the vCenter. The VMware Cloud Director should reflect the changes post a sync.
3. Increase the space on the flagged datastore to facilitate the relocation and thereafter re-attempt the delete operation. 

Additional Information

VMware Cloud Director 10.6 Release Notes, click here.