Virtual Machine deployment and power on fail in Tenant UI

Products

• VMware Cloud Director

Issue/Introduction

• Unable to perform deployment actions in tenant UI.
• Unable to power ON a vApp VM, the task fails with the below error after a long time and the VM stays powered OFF.
• Message that appears:
  
  [ ########-####-####-####-############ ] The vCenter Server operation ([vcId=########-####-####-####-############, moref=task-######) has taken too long (900000.00 seconds). VMware Cloud Director considers this operation timed out.
  
  
• In vCenter Server, there is no power ON task for the VM, only an Initialize powering On task.
• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, messages are seen as:

2023-03-17 16:56:23,415 | WARN | vim-proxy-activity-pool-23 | PowerOnVmActivity | [Activity Execution] Timed out while waiting for VMs [[vcId=UUID, moref=vm-####]] status to change to powered on in the inventory cache. Waited 300 seconds for the VMs status to change to > powered on. - Handle: urn:uuid:########-####-####-####-############ | requestId=<REQUEST_UUID>,request=POST https://vcloud.example.com/api/vApp/vm-<VM_UUID>/power/action/powerOn,requestTime=1679071877867,remoteAddress=<IP>:60159,userAgent=Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTM...,accept=application/*+xml;version 38.0.0-alpha vcd=<VDC_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_ACTIVITY_UUID>) activity=(com.vmware.ssdc.backend.PowerOnVmActivity,urn:uuid:<TASK_ACTIVITY_UUID>) activity=(com.vmware.vcloud.fabric.foundation.vm.impl.PowerOperationActivity,urn:uuid:<TASK_ACTIVITY_UUID>) activity=(com.vmware.vcloud.val.internal.impl.PowerOnVmActivity,urn:uuid:<TASK_ACTIVITY_UUID>)

Environment

VMware Cloud Director 10.x

Cause

This issue occurs when the DRS setting in vCenter is set to Manual.
VMware Cloud Director does not support the power on function of VMs when DRS automation level is set to Manual in vCenter.

Resolution

Change the vSphere DRS automation level from Manual to Partially Automated or Fully automated. 
For more details, please refer 'vSphere Configuration Requirements'  here -  VMware Cloud Director Hardware and Software Requirements.