Error: "There are insufficient CPU resources to complete the operation. Please contact the System Administrator." when powering on VMs in Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Tasks such as powering on a VM, changing placement policy, modifying allocation on Organization VDC(OVDC) fails.
• Error seen from the UI:
  [ ########-####-####-####-############ ] The operation could not be performed, because there are insufficient CPU resources.
 - The amount of CPU resource available in the parent resource pool is insufficient for the operation.
• The Organization VDC and the backing Provider VDC(PVDC) has sufficient resources.
• Not all Organization VDCs show the issue. 
• Reconnecting vCenter doesn't change the behavior.
• The PVDC backing the affected OVDC is backed by multiple clusters.
• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, entries of the following form are seen:
  ERROR    | compute-fabric-activity-pool2 | GenericVirtualResourcePool     | vrp OVDC-name (########-####-####-####-############):########-####-####-####-############ - Failed to reconfigure resource pool | ... activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-####-############) activity=(com.vmware.vcloud.valc.activities.UpdateVdcActivity,urn:uuid:########-####-####-####-############) activity=(com.vmware.vcloud.valc.activities.UpdateComputeActivity,urn:uuid:########-####-####-####-############)
com.vmware.ssdc.library.exceptions.InsufficientCpuResourcesException: The operation could not be performed, because there are insufficient CPU resources.
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:398)
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:500)
        at com.vmware.vcloud.val.internal.impl.VC20VirtualEngine.reconfigureResourcePool(VC20VirtualEngine.java:3035)
        at com.vmware.vcloud.val.internal.impl.VC20VirtualEngine.reconfigureResourcePool(VC20VirtualEngine.java:3062)
        at com.vmware.vcloud.fabric.compute.vrp.GenericVirtualResourcePool.updateResourcePool(GenericVirtualResourcePool.java:973)
        at com.vmware.vcloud.fabric.compute.vrp.GenericVirtualResourcePool.updateBackingForCapacityChange(GenericVirtualResourcePool.java:3019)
...
Caused by: (vim.fault.InsufficientCpuResourcesFault) {
   faultCause = null,
   faultMessage = null,
   unreserved = 570,
   requested = 39950
}
• For each cluster backing the PVDC, at the time of operation, the reservation is set to 0.


Environment

VMware Cloud Director

Cause

This is a known issue when the affected OVDC is backed by multiple PVDC resource pools. At the time of operation, the reservation on the secondary PVDC resource pool is set to 0 and the placement is carried out only on the primary resource pool. 

This issue can happen when concurrent tasks cause the VC inventory update component for VCD to slow down. 

Resolution

To permanently resolve the issue, upgrade to Cloud Director 10.6. You can download VMware Cloud Director 10.6 from the Broadcom Support Portal
Note: If you have VCDA, VCDA 4.7.3 is compatible with VCD 10.6.01. Please verify product and version compatibility before any upgrades using the Product Interoperability Matrix

As a workaround, set the OVDC to have unlimited CPU allocation.

Alternatively, increase the number of retry count for the placement. Update below config parameter to 10 using cell management tool (CMT). Default value is 5.

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n retry.count.to.consider.currentHub -l 
/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n retry.count.to.consider.currentHub -v 10

Note: It doesn't requires VCD restart.