HCX - Creating new Service Mesh fails at 60% with error "Interconnect Service Workflow OvfUpload failed. Error: null. Cause: null"

Products

• VMware HCX

Issue/Introduction

• Deployment Cluster has ESXi hosts with custom certificates
• After submitting the task for creating the Service Mesh, an error occurs that prevents the process from completing successfully. The error message is as follows:
  
  
  
• The following messages can be found in the log file /common/logs/admin/app.log:
  

  2025-01-14 16:01:36.253 UTC [InterconnectService_SvcThread-18819, J:ae675662, , TxId: ########-####-####-####-############] ERROR c.v.v.h.s.i.InitiateServiceMeshOperation- Process service mesh workflow failed. Error: Process Service Mesh failed.  Deploy and Configuration of Interconnect Appliances Failed. Interconnect Service Workflow DeployAppliance failed. Error: Interconnect Service Workflow OvfUpload failed. Error: null. Cause: null
  2025-01-14 16:01:36.260 UTC [InterconnectService_SvcThread-18802, J:ae675662, , TxId: ########-####-####-####-############] INFO  c.v.v.h.s.i.InitiateServiceMeshOperation- initiateServiceMeshOperation Running in state: FAILED for servicemesh-######-####-####-####-#############
  2025-01-14 16:01:36.268 UTC [FailureDetectionService_EventListener, , , TxId: ] INFO  c.v.v.h.f.FailedJobEventsListener- Uploading failure jobType: InterconnectServiceJobs workflowType: processServiceMesh failure: 7
  2025-01-14 16:01:36.304 UTC [FailureDetectionService_EventListener, , , TxId: ] ERROR c.v.v.h.f.PhConnection- Error uploading the job errors  to the Phone home com.vmware.vchs.hybridity.adapters.https.UntrustedCertificateException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

• From the HCX the command openssl s_client  -connect "ESXI FQDN:443" -showcerts -servername "ESXi Name" shows the following message:

  Verify return code: 21 (unable to verify the first certificate)

Cause

The issue occurs because the HCX Manager cannot verify the ESXi certificate chain, leading to a dropped TCP connection.

Resolution

• Make sure that the ESXi host certificate chain is valid
• Import the certificate from ALL ESXi hosts available in the deployment cluster the HCX Trusted CA Certificate page:
  
  1. Navigate to the appliance management interface: https://hcx-ip-or-fqdn:9443.
  2. Navigate to the Administration tab.
  3. Select Certificate > Trusted CA Certificate on the side menu.
  4. Select the certificate import option:URL
  5. Enter the ESXi URL.
     For example, https://"ESXi FQDN"
  6. Click Apply.

Additional Information

Importing Trusted Certificates from a Remote Site