HCX - Unable to deploy/redeploy HCX appliances - OVF upload failed

Products

• VMware HCX

Issue/Introduction

Whenever user attempts to deploy or redeploy HCX appliances they are met with the below error

Workflow OVFUpload failed. Error: Connection timed out

 

Error as seen in HCX-MGR app.log -- Found in /common/logs/admin:

2024-07-10 13:48:41.499 UTC [InterconnectService_SvcThread-29355, J:a428b267, , TxId: ########-####-####-####-b476637e3c25] ERROR c.v.v.h.s.i.InitiateApplianceOperation- UpgradeAppliance failed, errorCode:null. stacktrace:null, errorMessage:Interconnect Service Workflow InterconnectRedeploy failed. Error: Redeploy of Interconnect Appliance failed. Interconnect Service Workflow InterconnectRedeploy failed. Error:  Deploy and Configuration of Interconnect Appliances Failed. Interconnect Service Workflow DeployAppliance failed. Error: Interconnect Service Workflow OvfUpload failed. Error: Connection timed out (Connection timed out)

Verify the URL being provided by vCenter is also seen by HCX. 

• Find the NFC session ID used in HCX app.log:

1. 1. Cat app.log and grep for the TxId shown in error noted above.
      
      

      TxId: ########-####-####-####-b476637e3c25

      cat app.log | grep ########-####-####-####-b476637e3c25 | grep -i nfc 

       

      2024-07-10 13:45:18.923 UTC [OvfUploadService_SvcThread-20, Ent: HybridityAdmin, , TxId: ########-####-####-####-b476637e3c25] INFO  c.v.v.h.s.ovfupload.OvfUploadJob- Uploading file file:/common/appliances/vcc/4.9.1.0/VCCGateway-4.9.1.0-23822403-disk1.vmdk to https://vc-fqdn:400XX/nfc/########-####-####-####-3fd348d3ba1b/disk-0.vmdk
      

      
      
      
   2. Take note of the NFC session ID: ########-####-####-####-3fd348d3ba1b
   3. SSH to vCenter and grep for this session ID in vpxd.log -- Found in /var/log/vmware/vpxd
      
      
   4. cat /var/log/vmware/vpxd.log | grep ########-####-####-####-3fd348d3ba1b | grep -i nfc
      

      2024-07-10T13:45:18.496Z info vpxd[06745] [Originator@6876 sub=MoHttpNfcLease opID=########-####-####-####-b476637e3c25-34-01] Host URL: https://<ESXi-IP>OR<HOSTNAME>/nfc/########-####-####-####-3fd348d3ba1b/, target ID: disk-0.vmdk

You can see the FQDN or IP for the ESXi host chosen to receive this OVF can be verified via vpxd logs. And in this case HCX-MGR received the FQDN of vCenter instead along with a new port number (destination port NAT).

Environment

HCX 4.7-4.x

Cause

OVF upload can fail for a variety of reasons such as no connectivity to VC and or ESXi on required ports. 

HCX-MGR <-->vCenter : 443

HCX-MGR --> ESXi : 443, 80 & 902

The above example/scenario was caused by a Gateway appliance in between HCX and VC that was performing HTTPs traffic inspection and modifying the URL provided by vCenter to facilitate OVF upload to ESXi. In this case the Gateway appliance was modifying the https payload and stripping the ESXi host IP provided by VC and replacing it with the FQDN of vCenter along with the addition of a new port #. 

 

Resolution

Disable HTTP/HTTPS packet inspection on any networking devices inspecting and modifying traffic between the HCX and vCenter appliances.