vSAN Skyline Health tab is missing, vSAN Health Service fails, and vCenter Server services report HTTP 503 Service Unavailable errors
search cancel

vSAN Skyline Health tab is missing, vSAN Health Service fails, and vCenter Server services report HTTP 503 Service Unavailable errors

book

Article ID: 413676

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • The vSAN Skyline Health tab is missing from the cluster.
  • The vSAN Service section displays "No healthy upstream".
  • vCenter Server backup operations may fail and generate a corresponding alarm in vCenter Server.
  • The vmware-vsan service may be noticed to be abruptly stopped and failed to start with the following error: "An error occurred while starting service '%(0)s'",
  • Following error is reported in the /var/log/vmware/cloudvm/service-control.log:

    <>timestamp<> ERROR service-control Error executing start on service vsan-health. Details {
        "detail": [
            {
                "id": "install.ciscommon.service.failstart",
                "translatable": "An error occurred while starting service '%(0)s'",
                "args": [
                    "vsan-health"
                ],
                "localized": "An error occurred while starting service 'vsan-health'"
  • Validation of /var/log/vmware/vsan-health/vmware-vsan-health-service.log confirms that the vSAN Health Service is unable to communicate with internal vCenter services because requests are being rejected with HTTP 503 Service Unavailable errors.

2026-06-14T06:55:18.562Z ERROR vsan-mgmt[4023953] [VsanHealthHelpers::QueryVcCloudType opID=noOpId] Cannot get the cis service attribute: 503 Service Unavailable
http.client.HTTPException: 503 Service Unavailable

  • Validation of /var/log/vmware/vpxd/vpxd.log confirms that vCenter is unable to retrieve vSAN performance and health information because backend service requests are failing with HTTP 503 errors.

2026-06-14T06:49:25.746Z info vpxd[14561] SOAP request returned HTTP failure; method: queryVsanPerf; code: 503(Service Unavailable)
2026-06-14T06:49:25.746Z warning vpxd[14561] Exception was thrown when call vsan-performance-manager
HTTP error response: Service Unavailable

  • Validation of /var/log/vmware/vsan-health/vsanvcmgmtd.log confirms that internal vSAN services are unable to maintain authenticated communication, preventing health information from being updated successfully.

2026-06-14T08:06:40.352Z info vsanvcmgmtd[2252726] [vSAN@6876 sub=vmomi.soapStub[32] opId=7b12491d] SOAP request returned HTTP failure; <<io_obj p:0x00007f6f50583068, h:43,
 <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-##/vsan>, method: waitForVsanHealthGenerationIdChange; code: 500(Internal Server Error); fault: (vim.fault.NotAuthenticated) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    object = 'vim.host.VsanHealthSystem:ha-vsan-health-system',
-->    privilegeId = "",
-->    missingPrivileges = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f6f50583068, h:43, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-##/vsan>]: waitForVsanHealthGenerationIdChange
--> "
--> }

Environment

  • VMware vCenter 8.x
  • VMware VCF 5.x
  • VMware VCF 9.x

Cause

  • The issue is caused by a known vCenter Server issue where the Envoy Sidecar service exhausts its allocated memory and becomes overloaded.
  • When Envoy reaches its memory limit, it begins rejecting internal service requests with HTTP 503 Service Unavailable responses.
  • Since multiple vCenter services rely on Envoy for internal communication, the overload condition impacts authentication services, vSAN Health Services, backup status reporting, and vCenter management services, resulting in the observed alarms and login failures.
  • Validation of /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log confirms that the vSphere Client is unable to communicate with backend services because Envoy is overloaded and rejecting requests.

[2026-06-12T07:03:48.563Z] [ERROR] agw-httpclient-worker-3       c.v.v.plugin.notification.WcpServicePluginNotificationsListener   Error with subscription to vc hostname (6436973e-3676-4a49-####-########), nodeId: 5dc3e23c-e26f-4b6d-####-######### com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): envoy overloaded

  • Validation of /var/log/vmware/vpxd-svcs/vpxd-svcs.log confirms that internal vCenter services are failing because requests routed through Envoy are being rejected.

2026-06-13T00:03:17.482Z [VLSI-session-reaper [] WARN  com.vmware.vim.vmomi.server.session.impl.SessionImpl  opId=] Caught exception while closing session auth data
com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): envoy overloaded

  • Validation of /var/log/vmware/sso/websso.log confirms that authentication services are affected because Envoy is rejecting service requests due to overload.

2026-06-14T07:37:30.476Z INFO websso[46:tomcat-http--8] [CorId=dccb7262-e96c-4897-b5af-a0df48809d54] [com.vmware.identity.samlservice.impl.ExternalIdpProvider] Got exception (sleeping before retry)
com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): envoy overloaded

  • Validation of /var/log/vmware/envoy-sidecar/envoy-access-* confirms approximately 497,956 overload events, demonstrating that the Envoy Sidecar service is under sustained resource pressure and is actively rejecting requests.

root@vc_hostname[ / ]# zgrep "503 overload" /var/log/vmware/envoy-sidecar/envoy-access-* | wc -l
497956

Resolution

  • This issue is resolved in VMware vCenter Server 8.0 Update 3e
  • For vCenter Server deployed as part of VMware Cloud Foundation (VCF), the fix is not currently available and will be included in a future VMware Cloud Foundation release.
Powered by Wolken Software