Error: "Internal Server Error" when trying to power on Virtual Machines in Flex VDCs after upgrading VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• You are unable to power on Virtual Machines(VMs) in Flex VDCs after upgrading VMware Cloud Director(VCD) to 10.6.1.2 from 10.6.
• You can see the below errors from the below mentioned logs when you ssh to either of the cells for the VCD enviornment
  /opt/vmware/vcloud-director/logs/vcloud-container-debug.log:
  
  ERROR    | task-service-activity-pool-27 | FutureUtil                     | Failed to deploy VM | requestId=########-####-####-####-########8b12,request=POST https://,vcd-fqdn>/api/vApp/vm-########-####-####-####-########5c99/power/action/powerOn,requestTime=######,remoteAddress=##.##.##.###:#####,...
com.vmware.ssdc.util.LMException: Internal Server Error
        at com.vmware.ssdc.util.LMException.wrap(LMException.java:135)
        at com.vmware.ssdc.library.ExceptionFactory.createFromMultiple(ExceptionFactory.java:32)
        at com.vmware.ssdc.backend.DeployVAppVmSubsetActivity.checkFuturesForException(DeployVAppVmSubsetActivity.java:529)
        at com.vmware.ssdc.backend.DeployVAppVmSubsetActivity$DeployNetworksPhase.invoke(DeployVAppVmSubsetActivity.java:210)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
...
Caused by: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error
        at com.vmware.vcloud.common.future.FutureUtil.convertExecutionException(FutureUtil.java:264)
        at com.vmware.vcloud.common.future.FutureUtil.checkCompletedFuture(FutureUtil.java:203)
        at com.vmware.ssdc.backend.DeployVAppVmSubsetActivity.checkFuturesForException(DeployVAppVmSubsetActivity.java:522)
        ... 8 more
Caused by: java.lang.NullPointerException
        at java.base/java.util.ArrayList.addAll(ArrayList.java:702)
        at com.vmware.vcloud.fabric.compute.vrp.GenericVirtualResourcePool.addNewResourcePoolUnderElasticVdc(GenericVirtualResourcePool.java:4391)

Environment

VMware Cloud Director 10.6.1.2
VMware Cloud Director 10.6.1.1

Cause

This issue is caused because of few stale compute ID entries present in VCD DB which are already deleted from vCenter. This is the reason you will receive the above error of  Null Pointer Exception while accessing the VMs.

Resolution

The stale entries needs to be removed from VCD DB and database modification are needed. Please contact Broadcom Support and note this Article ID (427784) in the problem description. For more information, see Creating and managing Broadcom support cases.