Error: 'Internal Server Error' when trying to create any Virtual machine from templates in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• When you are trying to create a Virtual Machine(VM) using an old or newly deployed template in VMware Cloud Director (VCD) you can see the below error:
  
  com.vmware.ssdc.util.LMException: Internal Server Error
      at com.vmware.ssdc.util.LMException.wrap(LMException.java:135)
      at com.vmware.ssdc.library.ExceptionFactory.createFromMultiple(ExceptionFactory.java:32)
      at com.vmware.vcloud.vdc.impl.ComposeRecomposeVAppActivity$FinalizeRecomposePhase.invoke(ComposeRecomposeVAppActivity.java:1592)
      at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
      at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
  
  
• Tenants are unable to create any new Kubernetes Cluster as the Ephemeral VM never powers on.
• You will find the similar errors as have been mentioned below from the cse.logs:
  
  {"level":"info","ts":"yyyy-mm-ddThh:mm:ss.msZ","caller":"repair/heartbeat.go:120","msg":"Exiting heartbeat since watched thread [########-####-####-####-########9ae5] of RDE [<cluster-name>(urn:vcloud:entity:vmware:capvcdCluster:########-####-####-####-########0b3f)] has not pinged since [10 * 60]","workerID":"########-####-####-####-########9ae5"}
I0206 hh:mm:ss.ms 1582 gateway.go:70] Obtained Gateway [<Name of the Gateway>] for Network Name [<Name of the Routed Network>] of type [NSXT_FLEXIBLE_SEGMENT]
  
  
• You can also see the below error from /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on VCD cell:
  
  ERROR    | vim-proxy-activity-pool-106 | ReconfigureVmActivity          | [Activity Execution] Uncaught Exception during Activity execution. Recent phase: com.vmware.vcloud.val.internal.impl.ReconfigureVmActivity$ReconfigureVmInVcPhase@1f60a499 - Handle: urn:uuid:########-####-####-####-########0ff3, Current Phase: ReconfigureVmActivity$ReconfigureVmInVcPhase | requestId=########-####-####-####-########c13d,request=POST https://<vcd-fqdn>/api/vdc/########-####-####-####-########62a5/action/instantiateVmFromTemplate
...
(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-####-########dd03e) activity=(com.vmware.vcloud.vdc.impl.ComposeRecomposeVAppActivity,urn:uuid:########-####-####-####-########5ab0) activity=(com.vmware.vcloud.vdc.impl.CreateStoredVmActivity,urn:uuid:########-####-####-####-########6e86) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmNonBlockingActivity,urn:uuid:########-####-####-####-########2e5f) 
...
java.util.concurrent.ExecutionException: com.vmware.vcloud.val.exceptions.TaskNotificationException: Failed to receive status for task [vcId=########-####-####-####-########df50c, moref=task-####]
 at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:136)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:829)

Environment

VMware Cloud Director 10.6.1

Cause

This issue occurs when there are some inter-cell communication issues for the VCD enviornment. 

Resolution

This issue can be resolved by restarting the VCD services on all the cells. For steps to stop, start, or restart the VMware Cloud Director service you can follow the document: Steps to stop, start, or restart the VMware Cloud Director service before rebooting the nodes