Error: "com.vmware.ssdc.library.exceptions.VimTimedoutException: The VMware Cloud Director operation has timed out" when importing large OVFs

Products

• VMware Cloud Director

Issue/Introduction

• When trying to import large OVF or OVA files as a vApp or vApp Template into VMware Cloud Director, upload to transfer storage succeeds but when VMs are being uploaded to vCenter the task fails.
  
  
• Error can be seen in the Cloud Director Cells' /opt/vmware/vcloud-director/logs/vcloud-container-debug.log of the form:
  
  updateFailedJob(com.vmware.ssdc.library.exceptions.VimTimedoutException)
... 
Caused by: (vim.fault.Timedout) { 
   faultCause = null, 
   faultMessage = null
  
  
• Error can be seen in the vCenter Server's /var/log/vmware/vpxd/vpxd.log of the form:

  <timestamp> info vpxd[#######] [Originator@#### sub=vmomi.soapStub[######] opID=vcd-########-####-####-####-############;activity=urn:uuid:########-####-####-####-############-##] SOAP request returned HTTP failure; <<io_obj p:0x0000############, h:##, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-####/vpxa>, method: renewHttpTicket; code: 500(Internal Server Error); fault: (vmodl.fault.RequestCanceled) {
    -->    faultCause = (vmodl.MethodFault) null,
    -->    faultMessage = <unset>
    -->    msg = "Received SOAP response fault from [<<io_obj p:0x0000############, h:##, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-####/vpxa>]: renewHttpTicket
    --> "
    --> }

• Error from VPXA Log on the ESX host:

  <timestamp> info vpxa[####]: [Originator@#### sub=HttpNfcTicket opID=vcd-########-####-####-####-############;activity=urn:uuid:########-####-####-####-############-##] [HTTP-NFC-TICKET] Expiring tickets...
<timestamp> info vpxa[####]: [Originator@#### sub=HttpNfcTicket opID=vcd-########-####-####-####-############;activity=urn:uuid:########-####-####-####-############-##] [HTTP-NFC-TICKET] Scheduling expiry for timeout[1197] sec.
<timestamp> info vpxa[####]: [Originator@#### sub=HttpNfcTicket opID=vcd-########-####-####-####-############;activity=urn:uuid:########-####-####-####-############-##] [HTTP-NFC-TICKET] Expiry done
...
<timestamp> info vpxa[####]: [Originator@#### sub=HttpNfcTicket opID=########] [HTTP-NFC-TICKET-MGR] Not renewing ticket due to no progress: ########-####-####-####-############
<timestamp> info vpxa[####]: [Originator@#### sub=HttpNfcTicket opID=########] [HTTP-NFC-TICKET ########-####-####-####-############] Revoked

Environment

• VMware Cloud Director 10.6.1.x
• VMware vCenter Server 8.0.3
• VMware vCenter Server 7.0.3

Cause

This can occur when the NFC lease in vSphere used by Cloud Director to deploy the OVF and associated files is timing out due to the size and rate of transfer of the files.

Resolution

To resolve the issue the NFC time out can be increased as per the vCenter knowledge base article here:

Exporting a VM as OVF template to content library or deploying an VM from OVF library item results in timeout error and export/import fails

Example steps would be as follows:

1. In the vSphere client go to vCenter > Config > Advanced Settings.
   
   
2. Locate the NFC time out parameters:
   
   config.httpNfcTicket.timeoutPeriodMinute
config.httpNfcTicket.renewalPeriod

NOTE: If they are not already present then they need to be added.
   When these values are not present the vCenter Server will use the default values.
   

3. Increase the values to a larger number and retry OVF import.
   NOTE: The following table shows possible example values to set the timeouts to 60 minutes, but the actual required values will vary depending on the upload speed from Cloud Director Cells to the destination ESX hosts.

Name	Value
config.httpNfcTicket.timeoutPeriodMinute	60
config.httpNfcTicket.renewalPeriod	3600