HCX Connector or Cloud Manager fails to reach HCX Service
search cancel

HCX Connector or Cloud Manager fails to reach HCX Service

book

Article ID: 314196

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

Identify a known issue with connectivity from HCX Connector or Cloud Manager to HCX Service and provide a solution.

Symptoms:
In some cases, HCX Connector or Cloud Manager will generate a banner in the UI with the error messages below to indicate it has failed to reach connect.hcx.vmware.com for an extended period of time: 
The HCX Manager has failed to reach https://connect.hcx.vmware.com beyond the grace period. Restore this connection to resume HCX services


The HCX Manager is unable to reach https://connect.hcx.vmware.com. The connection is required for authorization, critical updates and support


Cause

The messaging service in the HCX Connector or Cloud Manager has reached the maximum number of open files.
The service will become unstable and will restart constantly. The condition can be validated as follows:
  • In the HCX AUI (TCP 9443), go to "Appliance Summary" and check if "Messaging Service" is restarting and cycling from "Running" to different states such as "Starting" from time to time.
  • If possible, SSH into the HCX Connector or Cloud Manager using admin account and run the command "systemctl status kafka". Check the start timestamp to see if the Kafka service keeps restarting. The status will cycle to "activating" from time to time.
$ systemctl status kafka

kafka.service - Kafka
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)
Active: activating (start-pre) since Tue 2021-08-31 05:24:38 GMT; 270ms ago
  • If SSH access is available, search for "error" in the Kafka log /common/logs/kafka/kafkaServer.out (support bundle collection may be not be available) and confirm the "Too many open files" condition:
$ grep ERROR /common/logs/kafka/kafkaServer.out

[2021-08-31 05:30:39,703] ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
at kafka.network.Acceptor.accept(SocketServer.scala:615)
at kafka.network.Acceptor.run(SocketServer.scala:544)
at java.lang.Thread.run(Thread.java:748)

Resolution

Upgrade HCX Connector or Cloud Manager to version 4.2.1 or later.

Workaround:
If the condition is confirmed, since it may block the option to upgrade, open a Support Request with VMware Global Support Services to get it resolved.

IMPORTANT: Network connectivity to connect.hcx.vmware.com must be verified and the issue validated prior to opening an SR since the most common causes for the failure to reach HCX Service are due to underlay network infrastructure problems like incorrect proxy configuration, FW blocking traffic, and basic routing.

 


Additional Information

Impact/Risks:
All communication to HCX Service will be disrupted:
  • Software downloads will be unavailable.
  • License and feature validation will fail.
  • Support bundle collection may not be available.
Once the system has remain in this state after the 7 day grace period, all configuration and migration workflows will not be serviced.
Network Extension services will remain active but no configuration changes can be made.

Powered by Wolken Software