HCX - "Proxy returns "HTTP/1.1 503 Service Unavailable" while deploying Service Mesh/Appliance

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

This resource is to inform about Service Mesh deployment failure condition and how to recover.

Symptoms:
HCX Service Mesh deployment may fail during OVF deployment stage if web proxy configuration is used in HCX Manager 9443 UI.
Below error/exception can be seen in the HCX app-engine logs:

2023-06-15 07:03:22.542 UTC [OvfUploadService_SvcThread-1, Ent: HybridityAdmin, , TxId: 9e5e4340-xxxx-xxxx-xxxx-1402e624a72b] ERROR c.v.v.h.s.ovfupload.OvfUploadJob- Error in uploading Ovf
java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 503 Service Unavailable"

Location of App Engine log:

• HCX Manager : /common/log/admin/app.log

Cause

HCX Manager uses proxy server configuration to send ALL outbound HTTPS requests to the proxy server configured in the environment.
When proxy server is configured, All local connections must be explicitly excluded from proxy operation. Proxy Exclusions field can be used to submit IP/FQDNs belongs to vCenter Server, ESXi Server, NSX Manager, IX Appliance/HBR server etc. Refer HCX User Guide for more information.

If HCX Manager won't be able to resolve FQDNs for vCenter & ESXi Servers, the HTTPS request will be sent to proxy server, when only IP Addresses are configured in the proxy exclusion list.

Note: If vCenter/ESXi Servers are having FQDN configured then HCX Manager must be able to resolve IP against those FQDNs through configured DNS Server.

During OVF deployment process, HCX Manager tries to reach ESXi Server part of the deployment cluster specified in the Compute Profile (CP). Above exceptions can be seen if there are any connectivity issues between HCX Manager & ESX/vCenter Servers.

Resolution

DNS resolution should be working to ensure HCX Manager is able to resolve IP Address against all ESXi Servers part of the deployment cluster including vCenter Server.
DNS Server should be checked and fix if needed.

Workaround:
As a potential workaround, static resolution against all ESXi & vCenter Server FQDNs can be configured in the HCX Manager until DNS issues are fixed.
Note: Apply the workaround only on HCX Connector OR Cloud Manager depending upon local connectivity issues.

Steps:

• Open HCX Admin console and switch to the root user.
• Open the /etc/hosts file using your editor of choice as follows:

vi /etc/hosts

• Add the lines below to the end of the file

X.X.X.X   ESXi1.example.net
Y.Y.Y.Y   ESXi2.example.net

• Save the file and test ping connectivity from HCX Manager to respective ESXi & vCenter Servers.
• Deploy Service Mesh Appliances accordingly.

Additional Information

Impact/Risks:

• It will impact Update/Redeploy/Resync for an existing Service Mesh.
• New Service Mesh deployment will also be impacted.
• No impact to existing Network Extension services.
• Migration Services will be impacted.