Error provisioning provisioning Ephemeral VM; unable to create VM when attempting to create Kubernetes cluster in Cloud Director Container Service Extension

Products

• VMware Cloud Director

Issue/Introduction

• Kubernetes cluster creation task fails with following errors in Cloud Director(VCD):
  
  [500:INTERNAL_SERVER_ERROR] - [ #####-#####-#####-#####-##### ] Unable to perform this action. Contact your cloud administrator.\n - Task failed, vc-id={VC_UUID, task-moref={ManagedObjectReference: type = Task, value = task-1282458, serverGuid = null}, error={Cannot connect to host.}]]]]] during cluster creation]}]
  
  
  
  
  
• In the CSE(Container Service Extension) logs /root/cse.log reports you see following errors:
  
  Ephemeral VM for cluster [vApp_Name(urn:vcloud:entity:vmware:capvcdCluster:#####-#####-#####-#####-#####)] during cluster creation: [error provisioning Ephemeral VM; unable to create VM [EPHEMERAL-TEMP-VM] in vApp [vApp_Name] : [error for adding TKG VM to vApp[vApp_Name]: [unable to issue call to create VMs with prefix [EPHEMERAL-TEMP-VM] in vApp [vApp_Name] with template [Template_Name]: [failed to wait for task [&{https://VCD_URL/api/task/2/#####-#####-#####-#####-##### application/vnd.vmware.vcloud.task+xml urn:vcloud:task:#####-#####-#####-#####-#####  task error Recomposed Virtual Application vApp_Name(#####-#####-#####-#####-#####) vdcRecomposeVapp com.vmware.vcloud  false 0xc00086c198  <nil> 0xc0001258c0 API Error: 500: [ #####-#####-#####-#####-##### ] Unable to perform this action. Contact your cloud administrator.\n - Task failed, vc-id={VC_UUID}, task-moref={ManagedObjectReference: type = Task, value = task-1282458, serverGuid = null}, error={Cannot connect to host.} 0xc000125b80 0xc000125c80 100 [ #####-#####-#####-#####-##### ] Unable to perform this action. Contact your cloud administrator. <nil>}] created to add new VM of name [EPHEMERAL-TEMP-VM]: [task did not complete successfully:  [500:INTERNAL_SERVER_ERROR] - [ #####-#####-#####-#####-##### ] Unable to perform this action. Contact your cloud administrator.\n - Task failed, vc-id={VC_UUID}, task-moref={ManagedObjectReference: type = Task, value = task-1282458, serverGuid = null}, error={Cannot connect to host.}]]]]]]
  
  
• The corresponding task in vCenter fails with errors similar to the following:
  
  [500:INTERNAL_SERVER_ERROR] - [ #####-#####-#####-#####-##### ] Unable to perform this action. Contact your cloud administrator.\n - Task failed, vc-id={VC_UUID, task-moref={ManagedObjectReference: type = Task, value = task-1282458, serverGuid = null}, error={Cannot connect to host.}]]]]] during cluster creation]}]
  
  
  

Environment

VMware Cloud Director 10.x
VMware Cloud Director Container Service Extension 4.x

Cause

This issue occurs when the ESXi host backing the VCD Organization VDC does not have access to catalog where the TKG templates reside

Resolution

If the Kubernetes templates catalog is hosted on a remote vSphere cluster than the one that is backing the Organization VDC, a clone operation will be triggered to Organization VDC cluster. To resolve this issue, enable ESXi host access to resources(catalog datastore) or create a local catalog in vSphere cluster backing the Organization VDC.
For more information, please refer Create Catalogs and Upload OVA Files under VMware Cloud Director Container Service Extension Server Prerequisites.