Virtual Machine deployment and Power ON tasks fail in VMware Cloud Director Tenants
search cancel

Virtual Machine deployment and Power ON tasks fail in VMware Cloud Director Tenants

book

Article ID: 321454

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

  • Unable to perform deployment actions in tenant UI.
  • vApp/VM Power On task either completes successfully in VMware Cloud Director(VCD) without reaching vCenter or times out after a long time with following error:

    com.vmware.ssdc.library.exceptions.VimTaskTimeoutException: The vCenter Server operation ([vcId=########-####-####-####-############, moref=task-######) has taken too long (900000.00 seconds). VMware Cloud Director considers this operation timed out. at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$WaitForInventory

  • When a VM is started, the process is aborted after approximately 15 minutes.
  • VCD cannot start/create a VM
  • In vCenter Server, there is no power ON task for the VM, only an Initialize powering On/Reconfigure virtual machine task is observed.
  • Power Off tasks may still complete successfully, and power off the VM in vCenter.
  • In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, messages are seen as:

2023-03-17 16:56:23,415 | WARN | vim-proxy-activity-pool-23 | PowerOnVmActivity | [Activity Execution] Timed out while waiting for VMs [[vcId=UUID, moref=vm-####]] status to change to powered on in the inventory cache. Waited 300 seconds for the VMs status to change to > powered on. - Handle: urn:uuid:########-####-####-####-############ | requestId=<REQUEST_UUID>,request=POST https://cloud.example.com/api/vApp/vm-<VM_UUID>/power/action/powerOn,requestTime=1679071877867,remoteAddress=<IP>:60159,userAgent=Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTM...,accept=application/*+xml;version 38.0.0-alpha vcd=<VDC_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_ACTIVITY_UUID>) activity=(com.vmware.ssdc.backend.PowerOnVmActivity,urn:uuid:<TASK_ACTIVITY_UUID>) activity=(com.vmware.vcloud.fabric.foundation.vm.impl.PowerOperationActivity,urn:uuid:<TASK_ACTIVITY_UUID>) activity=(com.vmware.vcloud.val.internal.impl.PowerOnVmActivity,urn:uuid:<TASK_ACTIVITY_UUID>)

Environment

  • VMware Cloud Director 10.6.x

Cause

  • This issue occurs when the DRS setting in the vCenter Server cluster is set to Manual mode.
  • VMware Cloud Director does not support the power on function of VMs when DRS automation level is set to Manual in vCenter Server.
  • VCD when attempting to Power on / Modify VM configurations, would attempt to check if it has enough control over the vCenter cluster to move the VM resources around, if required, however with DRS set to Manual, it loses this ability. Thus marks the task as failed on VCD. 

Resolution

Additional Information

Powered by Wolken Software