Deploying vApps or VMs from Catalog Templates in Cloud Director does not complete and the job continues to run indefinitely

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• When attempting to deploy a vApp or VM in Cloud Director from a Catalog Template, jobs do not complete and remain in a running state.
• Creating a new vApp VM not from a Catalog Template succeeds without issue.
• Objects which are part of the deployment including vApps and VMs stay in a Busy state in the Cloud Director UI and with the job shows as running.
• The Cloud Director logs, /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, stop logging entries for the job and the Cloud Director Cells appear to no longer continue with any actions for this job.
• The last log entry which describes the ongoing task in the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log file is similar to:
  
  TaskManager | Attempting remove of wait handle for task [vcId=<VCENTER_ID>, moref=task-5730109] | requestId=<REQUEST_ID> vcd=<VCD_ID>,task=<TASK_ID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_ACTIVITY_ID>) activity=(com.vmware.vcloud.vdc.impl.CreateStoredVmActivity,urn:uuid:<ACTIVITY_ID>) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity,urn:uuid:<CREATE_ACTIVITY_ID>)
  
  
• State of the new VM deployed from the template will be COPYING_CONTENTS in the Cloud Director database.
• The VM clone operations will have completed in vSphere and the new VM will be present.

Environment

VMware Cloud Director 10.x

Cause

This issue occurs when Cloud Director Cells in the environment are unable to access the shared transfer storage mounted at /opt/vmware/vcloud-director/data/transfer.

The transfer storage is verified in the event that a Template deployment would require an inter vCenter copy in the case where multiple vCenter Servers are managed by Cloud Director.

Even in instances where the Template copy occurs within a single vCenter, Cloud Director will still validate the transfer storage during the deployment task.

If the transfer directory is not mounted then the activity enters a state where it does not finish.

Resolution

To ensure that Cloud Director Cells can Read/Write to the directory, access to the shared transfer directory should be verified.

1. Confirm the details of the mount, /opt/vmware/vcloud-director/data/transfer, in fstab on the Cloud Director Cells by running this command:
   
   cat /etc/fstab
   
   
2. Confirm that listing the file systems mounted to the Cloud Director Cells completes and that the transfer directory, /opt/vmware/vcloud-director/data/transfer, is listed as mounted:
   
   df -h
   
   
   If this succeeds, attempt to navigate to the transfer directory and confirm that the contents can be listed:
   
   cd /opt/vmware/vcloud-director/data/transfer
   
   ls
   
   If the transfer directory is not available across all Cloud Director Cells in the cluster, this issue is on the NFS server side and must be rectified. 
   
   For details on the requirements for this transfer directory, see Cloud Director requirements when using NFS for the transfer server storage (2086127).
   
   
   Once access to the transfer directory is restored and access from all Cloud Director Cells is confirmed deployment tasks can be completed in the Cloud Director UI without issues.