Error: 'Internal Server Error' Unable to Power on VM from vCloud Director

Products

• VMware Cloud Director

Issue/Introduction

• After deployment of VMs they fail to power on with:

com.vmware.ssdc.util.LMException: Internal Server Error
  at com.vmware.ssdc.util.LMException.wrap(LMException.java:128)
  at com.vmware.ssdc.library.ExceptionFactory.createFromMultiple(ExceptionFactory.java:32)
  at com.vmware.ssdc.backend.DeployVAppVmSubsetActivity.checkFuturesForException(DeployVAppVmSubsetActivity.java:526)
  at com.vmware.ssdc.backend.DeployVAppVmSubsetActivity$DeployNetworksPhase.invoke(DeployVAppVmSubsetActivity.java:196)
  at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
  at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error
  at com.vmware.vcloud.common.future.FutureUtil.convertExecutionException(FutureUtil.java:233)
  at com.vmware.vcloud.common.future.FutureUtil.checkCompletedFuture(FutureUtil.java:194)
  at com.vmware.ssdc.backend.DeployVAppVmSubsetActivity.checkFuturesForException(DeployVAppVmSubsetActivity.java:519)
  ... 8 more
Caused by: com.vmware.vcloud.fabric.compute.ResourcePoolNotFoundException

Environment

VMware Cloud Director 10.5

Cause

• Resource pools in vCenter are incorrectly mapped to Cloud Director OR
• A new cluster was added to back up the Provider VDC, VMs were deployed into the new cluster/resource pool, but Cloud Director has no reference in vrp_rp Table for it 

Resolution

1. This resolution could need to create some modifications to the Cloud Director Database.
2. Run the statements below and capture the outputs to each within a text file.
3. Create a ticket with Cloud Director Support.
4. Ensure you create a Database backup prior to any modifications being carried out on the Database. 

 

Steps to be done

1. SSH to the Primary Cell of Cloud Director
2.  Connect to the Database

   sudo -i -u postgres psql vcloud

3. For Provider VDCs backed by multiple clusters, to get the list of the resource pools and its details:
   

   SELECT
   prov_vdc.name,
   computehub_set_computehub.id,
   computehub_set_computehub.is_primary,
   vc_computehub.rp_moref,
   resource_pool_inv.cr_moref as cluster_moref
   FROM prov_vdc
   inner join prov_vdc_logical_resource on prov_vdc_logical_resource.prov_vdc_id = prov_vdc.id
   inner join computehub_set_computehub on computehub_set_computehub.computehub_set_id = prov_vdc_logical_resource.fo_id
   inner join vc_computehub on vc_computehub.id = computehub_set_computehub.computehub_id
   left join resource_pool_inv on resource_pool_inv.moref = vc_computehub.rp_moref
   left join resource_pool_inv system_rp on system_rp.moref = vc_computehub.service_rp_moref
   where prov_vdc.name='<Provider VDC name>';

4. Check if the Organization VDC resource pools can be seen from the Database and vCenter :
   

   select
   ovdc.name as Org_vDC,
   ovdc.sub_rp_moref as ovdcmoref,
   vrp.id as OrgvRP,
   rpinv.moref as ovdcVCmoref,
   rpinv.parent_moref as PVDCRPmoref,
   vrp_rp.*
   from org_prov_vdc ovdc
   left join vrp_rp on vrp_rp.sub_rp_moref = ovdc.sub_rp_moref
   left join vrp on vrp.id = vrp_rp.vrp_id
   left join resource_pool_inv rpinv on rpinv.moref=ovdc.sub_rp_moref
   where ovdc.name = '<Organization VDC name>';

   
   
5. Verify the number of resource pools listed (This should match the number of resource pools backing the Provider VDC on step 4*). 
6. Note the value of the Organization VDC resource pools from resource_pool_inv table.
7. From Database, run:
   

   select moref,cr_moref,parent_moref,name from resource_pool_inv where parent_moref in ('<rp_moref from Step 2>') and name like '<Organization VDC name>%';

   For example, if there are 3 Provider VDC resource pools shared with the Organization VDCs, the statement would be:

   select moref,cr_moref,parent_moref,name from resource_pool_inv where parent_moref in ('rp_moref_1','rp_moref_2,'rp_moref_3') and name like '<Organization VDC name>%';