HCX - Service Disconnected

Products

VMware HCX VMware Cloud on AWS

Issue/Introduction

To provide the resolution when the above HCX disconnect error is being seen in the HCX UI.

Symptoms:
When logging into the HCX UI, you may see the error "The HCX service is disconnected. Restore your connection to the HCX server to proceed." All migrations between the two HCX sites could be impacted.

Environment

HCX

Cause

This error is observed when the HCX Manager/Connector is not able to reach the HCX Activation URL - https://connect.hcx.vmware.com. HCX will periodically make calls to this URL to ensure it can successfully connect. When HCX cannot connect to this URL, it will display this banner and impact HCX functionality.

Resolution

Launch an SSH session to the HCX Manager/Connector and ensure connectivity to the HCX Activation URL - https://connect.hcx.vmware.com. Use the below curl command to test:

curl -v -k https://connect.hcx.vmware.com

If the above curl command fails to connect, ensure firewall rules are created to allow HCX Manager connectivity to the HCX URL over port 443.

If you are using a proxy

curl -v -k https://connect.hcx.vmware.com -x proxy:port

Ensure all local connections are added as a proxy exclusion within the HCX VAMI (please see doc below for Local Connection examples), and the HCX URLs (https://connect.hcx.vmware.com && https://hybridity-depot.vmware.com) are added as an exclusion on the Proxy Server - Configure a Proxy Server.

If the above curl command is successful, please follow the steps below:

Login to the HCX VAMI on port 9443 (https://hcx_manager_ip:9443)
Restart both the App Engine and Web Engine services.

If further issues are being seen after validating everything above, please proceed with contacting Broadcom Support. When creating the SR, please provide the following information:

Output of the Curl command
Confirmation if a Proxy Server has been added in HCX
HCX Log Bundle from the Source side with Appliance and DB dump included
Business Impact

Additional Information

Impact/Risks:
Migrations from Source to Destination will be impacted until the HCX Service is restored.

Sample output with the proxy:

curl -k -v https://connect.hcx.vmware.com -x <Proxy server>:<Port>
*   Trying <Proxy Server>:<Port>...
* Connected to <proxy server>(XXX.XXX.XXX.XXX) port <port> (#0)
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to connect.hcx.vmware.com:443
> CONNECT connect.hcx.vmware.com:443 HTTP/1.1
> Host: connect.hcx.vmware.com:443
> User-Agent: curl/8.1.2
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Broadcom Inc.; CN=connect.hcx.vmware.com
*  start date: Jun  7 00:00:00 2024 GMT
*  expire date: Jun  6 23:59:59 2025 GMT
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: connect.hcx.vmware.com
> User-Agent: curl/8.1.2
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 21 Nov 2024 12:27:19 GMT
< Content-Type: text/html
< Content-Length: 167
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Thu, 21 Nov 2024 13:27:19 GMT
< Location: https://hcx.vmware.com/#/home
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Server: cloudflare
< CF-RAY: 8e60ac55bd223a86-FRA
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
* Connection #0 to host <Proxy Server> left intact