VM Power-On Fails with "Internal Server Error"

Products

• VMware Cloud Director

Issue/Introduction

 

• Attempts to power on an existing Virtual Machine or deploy a new Virtual Machine fail in the VMware Cloud Director (vCD) UI.

• The task fails almost immediately.

• In the vCD UI The error is: 

  java.util.concurrent.ExecutionException: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error

   

•  

  within the vcloud-container-debug.log, you observe an error stack trace similar to the following:

  Caused by: java.lang.NullPointerException
          at com.vmware.vcloud.fabric.storage.storedVm.impl.AbstractCreateStoredVmActivity$AbstractAddVmToVmGroupPhase.invoke(AbstractCreateStoredVmActivity.java:503)
          at com.vmware.vcloud.activity.executors.PhaseRunnable.run(PhaseRunnable.java:41)
          ...

 

Environment

10.x

Cause

This issue occurs when the dedicated "Service Resource Pool" backing a Provider vDC in vCloud Director has been manually deleted, renamed, or recreated directly within the underlying vCenter Server.

When a Provider vDC is created, vCD automatically creates a hidden resource pool in vCenter named System vDC (<provider_vdc_id>). vCD stores the Managed Object Reference (moref) of this resource pool in the service_rp_moref column of the vc_computehub database table.

If this pool is deleted in vCenter, vCD's database retains the stale moref. When a VM power-on is requested, vCD attempts to place the VM into this non-existent pool, resulting in a NullPointerException.

Resolution

To resolve this issue, you must recreate the missing Resource Pool in vCenter with the exact name vCD expects, force an inventory sync, and update the vCD database with the new moref.

Disclaimer: Always take a full backup of the vCloud Director database before performing manual SQL updates.

Step 1: Identify the Missing Resource Pool

Run the following query against the vCD database to identify which Provider vDC is missing its backing resource pool in vCenter:

SELECT 
    pvdc.name AS provider_vdc_name,
    pvdc.id AS prov_vdc_id,
    vcenter.name AS vcenter_name,
    pvdc.rp_moref AS parent_cluster_or_rp_moref,
    'System vDC (' || pvdc.id || ')' AS missing_rp_name
FROM prov_vdc pvdc
JOIN virtual_center vcenter ON pvdc.vc_id = vcenter.id
LEFT JOIN resource_pool_inv rpi 
    ON rpi.name = 'System vDC (' || pvdc.id || ')' 
    AND rpi.vc_id = pvdc.vc_id
    AND rpi.is_deleted = false
WHERE rpi.moref IS NULL;

Take note of the exact string in the missing_rp_name column.

Step 2: Recreate the Resource Pool in vCenter

1. Log in to the vSphere Client for the affected vCenter.

2. Navigate to the Cluster or Parent Resource Pool associated with the Provider vDC.

3. Create a new Resource Pool.

4. Name the new Resource Pool exactly as it appeared in the missing_rp_name column from Step 1 (e.g., System vDC (XXXXXXX)).

Step 3: Synchronize vCenter Inventory

1. Log in to the VMware Cloud Director Provider UI.

2. Navigate to Infrastructure Resources > vCenter Server instances.

3. Select the affected vCenter Server and click Reconnect (or Refresh).

4. Wait 2-5 minutes to allow vCD to pull the newly created Resource Pool into its resource_pool_inv database table.
   
   Note: Check the URL of your browser while selecting the newly created Resource pool and look at the value 'resgroup-XXXX'. This value should match the one in Step 4.

Step 4: Generate and Apply the Database Fix

Run the following query to automatically find the new Resource Pool moref and generate the required SQL UPDATE statement:

WITH HubMapping AS (
    SELECT 
        vcc.id AS computehub_id,
        pvdc.id AS prov_vdc_id,
        'System vDC (' || pvdc.id || ')' AS expected_rp_name,
        vcc.vc_id
    FROM prov_vdc pvdc
    JOIN vc_computehub vcc 
        ON pvdc.vc_id = vcc.vc_id 
        AND pvdc.rp_moref = vcc.rp_moref
)
SELECT 
    hm.expected_rp_name,
    rpi.moref AS new_service_rp_moref,
    'UPDATE vc_computehub SET service_rp_moref = ''' || rpi.moref || ''' WHERE id = ''' || hm.computehub_id || ''';' AS copy_and_run_this_fix
FROM HubMapping hm
JOIN resource_pool_inv rpi 
    ON rpi.name = hm.expected_rp_name 
    AND rpi.vc_id = hm.vc_id
    AND rpi.is_deleted = false;

 

1. Copy the exact UPDATE command outputted in the copy_and_run_this_fix column.

2. Execute the UPDATE command against the vCD database.

3. Attempt to power on the VM again. The operation should now complete successfully.