Error:"error provisioning Ephemeral VM; unable to create VM" When deploying a CSE cluster

Products

• VMware Cloud Director

Issue/Introduction

• CSE cluster creation fails with the below error:
  
  [error provisioning Ephemeral VM; unable to create VM [EPHEMERAL-TEMP-VM] in vApp [vApp-Name] : [error for adding TKG VM to vApp[vApp-Name]: [unable to issue call to create VMs with prefix [EPHEMERAL-TEMP-VM] in vApp [vApp-Name] with template [Catalog/Ubuntu 22.04 and Kubernetes v1.30.2+vmware.1]: [failed to wait for task [&{https://cloud.example.com/api/task/########-####-####-########2f76 application/vnd.vmware.vcloud.task+xml urn:vcloud:task:########-####-####-########2f76 task error Recomposed Virtual Application vApp-Name(########-####-####-########bc1d) vdcRecomposeVapp com.vmware.vcloud DATE TIME false <nil> <nil> 0xc00014ab80 API Error: 500: [ ########-####-####-########b784 ] Unable to perform this action. Contact your cloud administrator. 0xc00014ad80 0xc00014ae80 100 [ ########-####-####-########b784 ] Unable to perform this action. Contact your cloud administrator. <nil>}] created to add new VM of name [EPHEMERAL-TEMP-VM]: [task did not complete successfully: [500:INTERNAL_SERVER_ERROR] - [ ########-####-####-########b784 ] Unable to perform this action. Contact your cloud administrator.]]]]] during cluster creation

• The CSE Server logs /root/cse.log may present below errors::
  
    {"level":"info","ts":"DATE TIME","caller":"vcdkesdk/vcdKeRdeManager.go:142","msg":"updating component [{VcdKeServerError DATE TIME UTC m=+1116866.501717671  vApp-Name map[Detailed Error:Watched worker thread [########-####-####-########2d3c] exited for RDE [vApp-Name(urn:vcloud:entity:vmware:capvcdCluster:########-####-####-########afbe)]]}] in RDE: [urn:vcloud:entity:vmware:capvcdCluster:########-####-####-########afbe]","workerID":"########-####-####-########2d3c"}
    {"level":"error","ts":"DATE TIMEZ","caller":"repair/heartbeat.go:150","msg":"Watched worker thread [########-####-####-########2d3c] exited for RDE [vApp-Name(urn:vcloud:entity:vmware:capvcdCluster:########-####-####-########afbe)]","workerID":"########-####-####-########2d3c","stacktrace":"gitlab.eng.vmware.com/core-build/vcd-k8s-provider/src/repair.(*Watcher).Start\n\t/app/src/repair/heartbeat.go:150"}
  
  
• The cluster template is on a catalog on a different Organization VDC backed by a vCenter different from the one the VM is created on  different vCenter.

Environment

VMware Cloud Director 10.x
VMware Container Service Extension 4.x

Cause

Cloud Director takes more than 3-4 minutes to deploy the Ephemeral VM since the cluster template is on a catalog on a different Organization VDC backed by a vCenter different from the one the VM is created on  different vCenter. 

By default, Container Service Extension estimates 3-4 minutes to hear back any sort of responses from Cloud Director.

If the VMs are taking more than the estimated time to be created, Container Service Extension does not receive any updates until VM creation is completed and the thread will exit due to these timeouts.

Container Service Extension will mark the cluster in an error during creation state and will then update the Cloud Director UI to show the failed cluster creation.

The delay in creating the ephemeral vAPP and VM is 

Resolution

Creating a catalog  containing the cluster template on the Organization VDC where  the cluster has to be deployed on.