vSphere client inaccessible and vAPI endpoint degraded due to Envoy overload
search cancel

vSphere client inaccessible and vAPI endpoint degraded due to Envoy overload

book

Article ID: 384498

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vSphere client fails to load and shows either of the following errors as a banner message and does not loads the login page and shows a blank page.

[400] An error occurred while processing the authentication response from the vCenter Single Sign-On server. Details: Status: urn:oasis:names:tc:SAML;2.O:status:Requester, sub status: null.

An error occurred while fetching identity providers. Please try again later. If problem persists, contact your administrator.
no healthy upstream

     

       

  • vCenter workloads may fail.
  • SSO login failure might fail with the error as above.
  • All vCenter services can be found running, no core dumps generated. Issue can be temporarily resolved by a vCenter reboot.
  • Some services report healthy with warnings, see examples below
    • vAPI Endpoint
      • Failed to retrieve SSO settings
      • Failed to login in SSO
      • Failed to retrieve VIM service URI from Lookup Service
  • The License (vmware-cis-license), vAPI Endpoint (vmware-vapi-endpoint) and VMware vSphere Profile-Driven Storage (vmware-infraprofile) services go into degraded state [healthy with warnings]
  • vCenter websso log (/var/log/vmware/sso/websso.log) reports envoy overloaded messages

[YYYY-MM-DDTHH:MM] INFO websso[71:tomcat-http--33] [CorId=########-####-####-####-############] [com.vmware.identity.samlservice.impl.ExternalIdpProvider] Got exception (sleeping before retry) com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): envoy overloaded

  • vCenter ssoAdminServer.log (/var/log/vmware/sso/ssoAdminServer.log)  reports envoy overloaded.

[YYYY-MM-DDTHH:MM] ERROR ssoAdminServer [2338 : pool-2-thread-503] [OpId=########-####-####-####-############] [com.vmware.vcenter.tokenservice.providers.VcIdentityInfoProviderImpl] Failed to get identity provider matching domain VMwareID com.vmware.vapi.client.exception. TransportProtocolException: HTTP response with status code 503 (enable debug logging for details) : envoy overloaded

  • vCenter envoy-access.log (/var/log/vmware/envoy/envoy-access.log) reports 

[YYYY-MM-DDTHH:MM] info envoy[3214] [Originator@6876 sub=Default] [YYYY-MM-DDTHH:MM] POST /sdk 500 via_upstream - 308 317 gzip 4008 4007 0 <IP_address>:58010 HTTP/1.1 TLSv1.2 <IP_address>:443 127.0.0.1:58482 HTTP/2 - 127.0.0.1:8085 - "Login"
[YYYY-MM-DDTHH:MM] info envoy[3214] [Originator@6876 sub=Default] [YYYY-MM-DDTHH:MM] POST /sdk 500 via_upstream - 209 331 gzip 1 1 0  <IP_address>:58010 HTTP/1.1 TLSv1.2 <IP_address>:443 127.0.0.1:58482 HTTP/2 - 127.0.0.1:8085 - "Logout"
[YYYY-MM-DDTHH:MM] info envoy[3214] [Originator@6876 sub=Default] [YYYY-MM-DDTHH:MM] POST /sdk 500 via_upstream - 308 317 gzip 4009 4008 0  <IP_address>:51010 HTTP/1.1 TLSv1.2 <IP_address>:443 127.0.0.1:58438 HTTP/2 - 127.0.0.1:8085 - "Login"
[YYYY-MM-DDTHH:MM] info envoy[3214] [Originator@6876 sub=Default] [YYYY-MM-DDTHH:MM] POST /sdk 500 via_upstream - 209 331 gzip 1 1 0  <IP_address>:51010 HTTP/1.1 TLSv1.2 <IP_address>:443 127.0.0.1:58438 HTTP/2 - 127.0.0.1:8085 - "Logout"
[YYYY-MM-DDTHH:MM] info envoy[3214] [Originator@6876 sub=Default] [YYYY-MM-DDTHH:MM] POST /sdk 500 via_upstream - 308 317 gzip 4008 4008 0  <IP_address>:51994 HTTP/1.1 TLSv1.2 <IP_address>:443 127.0.0.1:58482 HTTP/2 - 127.0.0.1:8085 - "Login"
[YYYY-MM-DDTHH:MM] info envoy[3214] [Originator@6876 sub=Default] [YYYY-MM-DDTHH:MM] POST /sdk 500 via_upstream - 209 331 gzip 1 1 0  <IP_address>:51994 HTTP/1.1 TLSv1.2 <IP_address>:443 127.0.0.1:58482 HTTP/2 - 127.0.0.1:8085 - "Logout"

  • To further validate the issue, the following command needs to be run from a SSH session to the vCenter.

 zgrep "503 overload" /var/log/vmware/envoy-sidecar/envoy-access-* | wc -l

  • If the result is different than 0, then execute:
    • For vCenter 8.0U3:

zgrep envoy_server_memory_heap_size{} /var/cache/vmware-rhttpproxy/envoy-sidecar-stats/* | cut -d ' ' -f2|  sort -n | uniq | tail -1 | awk '{print $1 >= 1052266987}'

    • For vCenter 9.0:

zgrep envoy_overload_envoy_resource_monitors_fixed_heap_pressure /var/log/vmware/vstats/metrics/ENVOY_SIDECAR* | grep -v "# TYPE" | cut -d ' ' -f2|  sort -n | uniq | tail -1 | awk '{print $1 >= 98}'

    • If the above command returns 1, the envoy-sidecar memory limit has been reached

  • The currently configured memory heap size for envoy-sidecar service can be found in /etc/vmware-envoy-sidecar/config.yaml

cat /etc/vmware-envoy-sidecar/config.yaml | grep -C 2 1073741824

    typed_config:
      "@type": type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
      max_heap_size_bytes: 1073741824 # 1GB
actions:
    - name: "envoy.overload_actions.disable_http_keepalive"

  • In a VCF environment, you may see the error "Utilization details are not available" under Workload domains.

Environment

  • VMware vCenter Server 8.0 U3g and lower
  • VMware vCenter Server 9.0.0.0
  • VMware Cloud Foundation 5.x
  • VMware SDDC Manager 5.2.x

Cause

The envoy-sidecar service reaches its default hardcoded memory limit of 1 GB (1073741824 bytes). When memory consumption reaches 98%, Envoy begins rejecting requests with an HTTP 503 envoy overloaded response. This is often triggered by high-frequency HTTPS session authentication requests from external monitoring tools or third-party services.

Resolution

  • If the issue is identified as multiple HTTPS session authentication requests to envoy and VC is running above 8.0 u3g. Identify what application/service refers to the <IP_address> and engage the respective vendor or team to address the issue. 
  • Incase if the issue is identified as known issue on vCenter running on 8.0 u3g and lower version, note that the issue is resolved in the following releases:
  • For vCenter 8.x, issue is resolved in 8.0 U3h. Log in to the Broadcom Support Portal to download this patch.
  • For vCenter 9.x, issue is resolved in 9.0.1.0. Log in to the Broadcom Support Portal to download this patch, depending on the entitlement VMware vSphere Foundation or VMware Cloud Foundation.

Workaround:

  1. Take snapshots of the vCenter or ELM vCenter group (See vCenter in Enhanced Linked Mode pre-changes snapshot best practice for guidance on taking snapshots of vCenters in ELM)

  2. Log in to the vCenter via SSH.
  3. Create a backup of the envoy sidecar config file:cp /etc/vmware-envoy-sidecar/config.yaml /etc/vmware-envoy-sidecar/config.yaml.back
  4. Using sed update the Envoy memory limit from 1073741824 (1 GB) to 2147483648 (2 GB): sed -i 's/max_heap_size_bytes: 1073741824/max_heap_size_bytes: 2147483648/g' /etc/vmware-envoy-sidecar/config.yaml
  5. Restart envoy-sidecar: service-control --restart envoy-sidecar
  6. Some cases have shown that Envoy memory limit of 2GB is not sufficient. In such cases, update the Envoy memory limit from 2 GB to 4 GB:

sed -i 's/max_heap_size_bytes: 2147483648/max_heap_size_bytes: 4294967296/g' /etc/vmware-envoy-sidecar/config.yaml

service-control --restart envoy-sidecar

    7. In some corner cases Envoy memory limit of 4 GB may not be enough. We recommend to completely remove the following two envoy-overload-actions from /etc/vmware-envoy-sidecar/config.yaml file.

name: "envoy.overload_actions.stop_accepting_requests"
      triggers:
        - name: "envoy.resource_monitors.global_downstream_max_connections"
          threshold:
            value: 0.99
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.98

 
  - name: "envoy.overload_actions.reject_incoming_connections"
      triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 1.00

   8.  Edit the envoy sidecar configuration file using vi editor to remove the two envoy-overload-actions (envoy.overload_actions.stop_accepting_requests and envoy.overload_actions.reject_incoming_connections):

vi /etc/vmware-envoy-sidecar/config.yaml

      a. After the two envoy-overload-actions are removed, the entire section for overload_manager in the /etc/vmware-envoy-sidecar/config.yaml file should look like this:

overload_manager:
  refresh_interval: 1s
  resource_monitors:
    - name: "envoy.resource_monitors.global_downstream_max_connections"
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
        max_active_downstream_connections: 8000
    - name: "envoy.resource_monitors.fixed_heap"
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
        max_heap_size_bytes: 4294967296 # 4GB
  actions:
    - name: "envoy.overload_actions.shrink_heap"
      triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.75
    - name: "envoy.overload_actions.disable_http_keepalive"
      triggers:
        - name: "envoy.resource_monitors.global_downstream_max_connections"
          threshold:
            value: 0.8
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.95
    - name: "envoy.overload_actions.reduce_timeouts"
      triggers:
        - name: "envoy.resource_monitors.global_downstream_max_connections"
          scaled:
            scaling_threshold: 0.25
            saturation_threshold: 0.97
        - name: "envoy.resource_monitors.fixed_heap"
          scaled:
            scaling_threshold: 0.85
            saturation_threshold: 0.97
      typed_config:
        "@type": type.googleapis.com/envoy.config.overload.v3.ScaleTimersOverloadActionConfig
        timer_scale_factors:
          - timer: HTTP_DOWNSTREAM_CONNECTION_IDLE
            min_timeout: 2s

      b. Save the file and exit (press ESC, type :wq!, press Enter)

      c. Restart envoy sidecar service:

          service-control --restart envoy-sidecar

Additional Information

  1. Occasionally, the vSAN menu remains missing from the vSphere UI despite the workaround. In these instances, the following logs will still be generated.
  2. Log location: /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

[YYYY-MM-DDTHH:MM] [ERROR] sdk-plugin-deployer-3140      com.vmware.vise.plugin.status.RemotePluginStatusServiceImpl       DEPLOYMENT_FAILED: Error deploying plugin package com.vmware.vsan.client:8.0.203.10000. Reason: Plugin configuration with Reverse Proxy failed.
[YYYY-MM-DDTHH:MM] [WARN ] sdk-plugin-deployer-104       com.vmware.bifrost.bus.EventBusLowApiImpl                         Failed to send message. Cannot find channel: plugin-state-change-notification
[YYYY-MM-DDTHH:MM] [INFO ] sdk-plugin-deployer-106       com.vmware.vise.plugin.registry.VcExtensionStateRegistry          Updating entry: Plugin: 'com.vmware.vsan.client:8.0.203.10000', State: 'FAILED_CONFIGURE'

    3. Then restart vmware-vsan-health and vsphere-ui services can resolve the problem.

#service-control --restart vmware-vsan-health 

#service-control --restart vsphere-ui 

Powered by Wolken Software