"no healthy upstream" error in vSphere Client and multiple service startup failures after rebooting vCenter Server 7.0

Products

• VMware vCenter Server

Issue/Introduction

After rebooting the vCenter Server, when accessing it via a browser, a "no healthy upstream" error is displayed, even though sufficient time has passed since the reboot.

When this issue occurs, communication failures between the envoy and rhttpproxy services are recorded in the following logs.

 - /var/log/vmware/rhttpproxy/rhttpproxy-##.log: 

YYYY-MM-DDThh:mm:ss:uuu error rhttpproxy[####] [Originator@6876 sub=EnvoyAds] Envoy rejected update. Error code: 13, message: Error adding/updating listener(s) sidecar_v4: route: unknown cluster 'sdkTunnel:8089'
--> edge_https_v6: route: unknown cluster '/websso/SAML2/SSOCAC'
--> edge_https_v4: route: unknown cluster '/websso/SAML2/SSOCAC'
--> edge_http_v6: route: unknown cluster 'sdkTunnel:8089'
--> edge_http_v4: route: unknown cluster 'sdkTunnel:8089'

 - /var/log/vmware/envoy/envoy-##.log: 

YYYY-MM-DDThh:mm:ssZ warning envoy[####] [Originator@6876 sub=config] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) sidecar_v4: route: unknown cluster 'sdkTunnel:8089'
edge_https_v6: route: unknown cluster '/websso/SAML2/SSOCAC'
edge_https_v4: route: unknown cluster '/websso/SAML2/SSOCAC'
edge_http_v6: route: unknown cluster 'sdkTunnel:8089'
edge_http_v4: route: unknown cluster 'sdkTunnel:8089'

 

 

Note: The "no healthy upstream" error can occur due to various reasons other than this issue. For more details, refer to "no healthy upstream" error when attempting to access the vCenter Server vSphere Client.

Environment

VMware vCenter Server 7.0

Cause

This race condition between the envoy and rhttpproxy services during startup prevents successful communication.
As a result, some services of vCenter Server (such as vpxd-svcs etc...) fail to start.

Resolution

This issue is resolved in VMware vCenter Server 8.0.

 

Workaround

To work around this issue and start the services correctly, run the following command to restart all vCenter Server services:

service-control --stop --all && service-control --start --all