HCX - Service Disconnected
search cancel

HCX - Service Disconnected

book

Article ID: 323664

calendar_today

Updated On:

Products

VMware HCX VMware Cloud on AWS

Issue/Introduction

  • When logging into the HCX UI, you see the error "The HCX service is disconnected. Restore your connection to the HCX server to proceed."

  • All migrations between the two HCX sites are impacted.  

Environment

HCX 4.10.x or lower.

Cause

  • This error is observed when the HCX Manager/Connector is not able to reach the HCX Activation URL -  https://connect.hcx.vmware.com.
  • HCX will periodically make calls to this URL to ensure it can successfully connect. When HCX cannot connect to this URL it will impact the HCX operations.
  • HCX might have a proxy server configured.

Resolution

  • Launch an SSH session to the HCX Manager/Connector via admin user and ensure there is a connectivity to the HCX Activation URL - https://connect.hcx.vmware.com.
  • Use the below curl command to test:
curl -v -k https://connect.hcx.vmware.com
  • If the above curl command fails to connect, ensure firewall rules are created to allow HCX Manager connectivity to the HCX URL over port 443. 

 

  • If you are using a proxy, then run the command below 
curl -v -k https://connect.hcx.vmware.com -x <proxy-server>:proxy-server-port

Note :- In the above command, proxy-server is the FQDN of the proxy server or the proxy server IP address.

  • The proxy server settings can be viewed via HCX Admin UI page https://<hcx-manager-ip-fqdn>:9443
  • Go to Administration --> Network Settings --> Proxy 

 

Note :- For HCX to function correctly when a proxy server is configured, local connections must be explicitly excluded from proxy operation. Use the Proxy Exclusions field. To know more details about this option, refer to the tech article here --> Configure Proxy Server on HCX

  • If the above curl command is successful, please follow the steps below:
    • Login to the HCX VAMI on port 9443 (https://hcx_manager_ip:9443)
    • Restart both the App Engine and Web Engine services. Refer to the article to restart the HCX services KB328973

If further issues are being seen after validating everything above, please proceed with contacting Broadcom Support. When creating the SR, please provide the following information:

    • Output of the Curl command
    • Confirmation if a Proxy Server has been added in HCX
    • HCX Log Bundle from the Source side with Appliance and DB dump included. To gather support bundle logs, refer to the page Generate HCX Support Bundle
    • Business Impact

 

 

Additional Information

Impact/Risks:
Migrations from Source to Destination will be impacted until the HCX Service is restored.

Sample output with the proxy:
curl -k -v https://connect.hcx.vmware.com -x <Proxy server>:<Port>
*   Trying <Proxy Server>:<Port>...
* Connected to <proxy server>(XXX.XXX.XXX.XXX) port <port> (#0)
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to connect.hcx.vmware.com:443
> CONNECT connect.hcx.vmware.com:443 HTTP/1.1
> Host: connect.hcx.vmware.com:443
> User-Agent: curl/8.1.2
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Broadcom Inc.; CN=connect.hcx.vmware.com
*  start date: Jun  7 00:00:00 2024 GMT
*  expire date: Jun  6 23:59:59 2025 GMT
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: connect.hcx.vmware.com
> User-Agent: curl/8.1.2
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 21 Nov 2024 12:27:19 GMT
< Content-Type: text/html
< Content-Length: 167
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Thu, 21 Nov 2024 13:27:19 GMT
< Location: https://hcx.vmware.com/#/home
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Server: cloudflare
< CF-RAY: 8e60ac55bd223a86-FRA
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
* Connection #0 to host <Proxy Server> left intact
Powered by Wolken Software