VM does not power on from vcd with error " Internal Server Error - [vcId=XXXX, moref=vm-XXXX] not on same VC as [vcId=XXXX, moref=resgroup-XXXX]"

Products

• VMware Cloud Director

Issue/Introduction

• While powering VM from vcd, power on fails with error in title
• vcloud-container-debug.log shows:
  

  (...)DeployVmOnFabricsActivity,urn:uuid:XXXXXXXXXXXXXXX)
   java.lang.IllegalArgumentException: [vcId=XXXXXX, moref=vm-XXXX] not on same VC as [vcId=XXXXXX, moref=resgroup-XXXX](...)

Environment

10.3.3

Cause

The error states that is not possible to move the VM vm-XXXX to the correct resource pool, while powering on since resgroup-XXXX belongs to a different vcenter ID

This can be caused by incorrect migration or move of VMs that stay on old places inside vcd database

Resolution

Read 'Additional Information' section before running these steps

From the logs collect bellow information:

• VM moref (moref=vm-XXXX)
• VM resource pool (moref=resgroup-XXXX)
• vcId from source and destination

Find inside vcd database where is this VM moref located:

select name,moref,resource_pool_moref,vc_id from vm_inv where moref='vm-XXXX';

Notes:

• 2 different VMs will be listed with same moref, on different resource pools and different vcenters exactly as the log error shows
• Match the name of your VM with the correct resource_pool (it will be different from the one collected from logs)

With this value you will need to find the provider VDC running this command:

select name from prov_vdc where id in (select prov_vdc_id from org_prov_vdc where sub_rp_moref='resgroup-XXXX');

With the name of the provider vdc from the previous command, you can now list all details from this Provider, filtered by it's Organization VDC: (replace ProviderVDC_NAME)

select
ovdc.name as Org_vDC,
vrp.id as vrp_id_vcd,
rpinv.moref as sub_rp_moref,
rpinv.parent_moref as cluster_moref,
vrp_rp.*
from org_prov_vdc ovdc
left join vrp_rp on vrp_rp.sub_rp_moref = ovdc.sub_rp_moref
left join vrp on vrp.id = vrp_rp.vrp_id
left join resource_pool_inv rpinv on rpinv.moref=ovdc.sub_rp_moref
where ovdc.name in (select name from org_prov_vdc where prov_vdc_id in (select id from prov_vdc where name like 'ProviderVDC_NAME'));

The output will list:

• Organization VDCs inside this Provider VDC
• vrp_id from VCD and from vcenter (Make sure it matches. This will be the ID of your resource pool named in vcenter as per your organization VDC. Each organization VDC will have a different ID)
• sub_rp_moref (find your resource pool here as resgroup-XXXX)
• computehub_id (This will be the ID for your provider VDC cluster and will be the same for every single organization and virtual machine inside it)
• sub_rp_vc_id (This is the ID of the vcenter. According to logs this will flag a different)

Now you know the correct computehub_id, vrp_id and sub_rp_moref so keep it in your notes.

List the current (incorrect information) with this command: (use the correct VM moref you got on previous step)

select deployment_status,id,vmmoref,computehub_id,vrp_id,vc_id from computevm where vmmoref='vm-XXXX';

This output will show:

• Our VM has a different computehub_id, vrp_id and possible the deployment_status is set to UNDEPLOYED 
• To fix this run the following statement:

update computevm set vrp_id='Correct_vrp_id',computehub_id='Correct_compute_hub_id',deployment_status='DEPLOYED' where id='Replace_with_id_from_previous_step';

Test the Power on

 

Additional Information

• Editing a database is risky, specially, if the db is live
• Before proceeding take backups and/or snapshots of all the VCD cells
• To access vcd database:
• SSH into primary cell
• Run the following command:
  

  sudo -i -u postgres psql vcloud 

• To exit from the database type:
  

  \q

• If in doubt, contact Broadcom Support before making changes