"The operation could not be performed, because there are insufficient resources" error causes VM power on to fail in Cloud Director when DRS Migration Threshold is set to fully Conservative

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Attempting to power on a VM in Cloud Director fails.
• The power on succeeds if performed directly in vCenter.
• In vSphere the DRS Migration Threshold is set to fully Conservative on the ESXi cluster where the VM is located.
• The power on tasks fail with an error in the Cloud Director cell /opt/vmware/vcloud-director/logs/vcloud-container-debug.log similar to:

| ERROR    | vim-proxy-activity-pool-123 | FutureUtil                     | Poweron VM task failed, vm-moref=<VM_MOREF>,vc-id=<VCENTER_UUID>  | requestId=<REQUEST_UUID>,request=POST https://<VCD_URL>/api/vApp/vm-<VM_UUID>/power/action/powerOn,requestTime=<REQUEST_TIME>,remoteAddress=<CLIENT_ADDRESS>,userAgent=<USER_AGENT>,accept=application/*+xml;version 32.0 vcd=<VCD_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_UUID>) activity=(com.vmware.ssdc.backend.PowerOnVmActivity,urn:uuid:) activity=(com.vmware.vcloud.fabric.foundation.vm.impl.PowerOperationActivity,urn:uuid:) activity=(com.vmware.vcloud.val.internal.impl.PowerOnVmActivity,urn:uuid:)
com.vmware.vcloud.val.exceptions.TaskManagerException: Task failed, vc-id={<VCENTER_UUID>}, task-moref={ManagedObjectReference: type = Task, value = <TASK_MOREF>, serverGuid = null}, error={The host does not have sufficient CPU resources to satisfy the reservation.}
        at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$InitialPhase.invoke(AsynchronousTaskWaitActivity.java:129)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.ssdc.library.exceptions.InsufficientUnknownResourcesException: The operation could not be performed, because there are insufficient resources.
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:401)
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:499)
        at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity.getResultIfTaskAlreadyCompleted(AsynchronousTaskWaitActivity.java:449)
        at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$InitialPhase.invoke(AsynchronousTaskWaitActivity.java:123)
        ... 7 more
Caused by: (vim.fault.InsufficientHostCpuCapacityFault) {
   faultCause = null,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         dynamicType = null,
         dynamicProperty = null,
         key = com.vmware.drs.constraints.cpu.InsufficientHostCpuCapacityFault,
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               dynamicType = null,
               dynamicProperty = null,
               key = requested,
               value = <REQUESTED_MHZ>
            },
            (vmodl.KeyAnyValue) {
               dynamicType = null,
               dynamicProperty = null,
               key = unreserved,
               value = <UNRESERVED_MHZ>
            },
            (vmodl.KeyAnyValue) {
               dynamicType = null,
               dynamicProperty = null,
               key = host,
               value = <HOST_MOREF>
            }
         ],
         message = The host <HOST_MOREF> can not satisfy the requested CPU resources of <REQUESTED_MHZ> MHz. Available CPU resources on the host: <UNRESERVED_MHZ> MHz.
      }
   ],
   host = ManagedObjectReference: type = HostSystem, value = <HOST_MOREF>, serverGuid = null,
   unreserved = <UNRESERVED_MHZ>,
   requested = <REQUESTED_MHZ>
}

Environment

VMware Cloud Director 10.x

Cause

This is a known issue in Cloud Director 10.3.x where powering on Cloud Director VMs will fail if the Provider VDC resource pools are backed by ESXi host clusters where the DRS Migration Threshold is set to fully Conservative.

Resolution

This issue is resolved in Cloud Director 10.3.3.4 which is available here, Download VMware Cloud Director.