vApp deployments are stuck in "UNRESOLVED" state

Products

• VMware Cloud Director

Issue/Introduction

• While trying to deploy new vApps from VCD UI, it does not proceed and is stuck in "UNRESOLVED" state.
• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log you can see error as below:
  | ERROR | task-service-activity-pool-<pool_id> | FutureUtil | Task future is not complete. | requestId=########-####-####-####-########,request=POST https://<vcd.example.com>/api/vApp/vapp-########-####-####-####-########/action/recomposeVApp,req
uestTime=<timestamp>,remoteAddress=##.##.##.##:31753,userAgent=python-requests/2.3.0 CPython/2.7.6 Linux/3.2.0-26-generic,accept=application/*+xml;version 37.0 vcd=########-####-####-####-########,task=########-####-####-####-######## activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:########-####-####-####-########)
com.vmware.ssdc.library.exceptions.VimTaskTimeoutException: The vCenter Server operation ([vcId=########-####-####-####-########, moref=task-4498370])
has taken too long (28800000.00 seconds). VMware Cloud Director considers this operation timed out.
        at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$WaitForInventoryUpdatePhase.invoke(AsynchronousTaskWaitActivity.java:313)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
  
  

 

Environment

• VMware Cloud Director 10.x

Cause

• This issue occurs when VMware Cloud Director loses connectivity to the vCenter server, causing the vApp deployment tasks to time out.

Resolution

• To resolve the underlying network connectivity issues between vCenter and VMware Cloud Director refer the article and perform the following validations:
  • Verify Network Connectivity:
    From each VCD cell, ping the vCenter server to ensure basic network connectivity is functional.
  • Check Port Availability:
    Confirm that port 443 is open between the VCD cells and vCenter by running the following command:
    curl -v telnet://<vCenter-IP>:443
  • Reconnect vCenter:
    After verifying connectivity, re-establish the connection to vCenter by following the steps outlined in the VMware documentation: Reconnect a vCenter Instance to Your VMware Cloud Director.
    Note : A reconnect with vcenter would cause temporary downtime with any ongoing tasks. 
    
    
• Additionally, If the issue is noticed with only specific vApps, and if a temporary downtime is feasible, we could follow through the below:
  
  • Find a VM in <Inconsistent> state.
  • Under General tab for the affected VM note the <Name in vSphere>.
  • Log in to the vCenter Server.
  • Find the VM with the name found in step 2.
  • Right-click on the VM in vCenter Server and click Power > Shutdown Guest OS or Power Off.
  • In the VCD UI, on the affected VM Click on All actions > Power >  Power On.