All hosts show NO_RESPONSE after upgrading vCenter to 9.1 in VCF Environment due to envoy-hgw service crash

Products

• VMware vCenter Server

Issue/Introduction

• After upgrading vCenter Server to version 9.1 in a VMware Cloud Foundation (VCF) environment, all ESXi hosts appear as NO_RESPONSE.
• Attempts to reconnect hosts fail with: "A general system error occurred: Error: system_error Mussages: vapi.sund.failed<Send of frame failed: N7Vmacore15SysterrExceptionE(Connection refused: The remote service is not running, OR is overloaded. OR a firewall is rejecting connections.)".
• The envoy-hgw service is crashed quickly after starting with trace .
  CRITICAL envoy-hgw 81562 [vc@4413 sub="main"] std::terminate called! Uncaught EnvoyException 'Failed to read BIO X509', see trace.
• The crash trace in the envoy logs typically ends with a specific ESXi IP address (e.g., associated with an HCX-IX host).
   ConnectionInfoSetterImpl 0x1324bf891550, remote_address_: <ESXi IP>:443, direct_remote_address_: <ESXi IP>:443, local_address_: <vCenter IP>:42722, server_name_:

Environment

VMware Cloud Foundation (VCF) 9.1
vCenter Server 9.1

 

Cause

The envoy-hgw (Host Gateway) service crashes when it encounters a specific misconfigured or problematic IP address during the SSL handshake/certificate validation process. This crash breaks the communication path between vCenter and all managed ESXi hosts.

Resolution

1. Access the vCenter Server via SSH.
2. Review the envoy-hgw logs to identify the specific IP address mentioned in the final ConnectionInfoSetterImpl trace.
3. Check the Host with that IP and Remove the Host from vCenter Inventory
4. Restart the envoy-hgw service on vCenter: service-control --restart vmware-envoy-hgw
5. Verify that the service remains running and hosts transition back to a Connected state.
6. Fix the issue on the removed host