Istio Ingress Gateway Returning HTTP 431 – Request Header Fields Too Large

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Applications exposed through the Istio Ingress Gateway may fail with the following HTTP response:

Status Code: 431
Error: Request Header Fields Too Large

• In this scenario, the request fails at the ingress gateway level and never reaches the backend service. The response is generated by the Envoy Proxy running within the Istio ingress gateway when the request headers exceed the allowed size.
  
  
• This issue is typically observed when accessing applications through a browser or API clients where the request contains large cookies, authentication tokens, or multiple forwarded headers.

• Verify Request Header Size
  • Inspect request headers from a client using curl:

curl -kv https://<application-url>

• Review the output for large headers such as:

  • Cookie

  • Authorization

  • X-Forwarded-For

• The header size can also be estimated using:

curl -s -D - https://<application-url> -o /dev/null | wc -c

• If the size approaches or exceeds 60 KB, Envoy may reject the request.

Environment

• VMware vSphere Kubernetes Service

Cause

• The Istio ingress gateway relies on Envoy to process incoming traffic. Envoy enforces limits on the total size of HTTP request headers.
• By default, Envoy allows a maximum request header size of approximately 60 KB. If the cumulative size of all request headers exceeds this threshold, Envoy immediately rejects the request with an HTTP 431 response.
• Common factors that can lead to oversized request headers include:

  • Large Cookie headers generated by web applications

  • Authentication tokens such as JWT in the Authorization header

  • Single Sign-On (SSO) or identity provider cookies (OIDC, OAuth, SAML)

  • Multiple proxies adding X-Forwarded-* headers

  • Applications storing session data directly in cookies

• As the ingress gateway rejects the request during header validation, the backend application pods never receive the request.

Resolution

Option 1 – Increase the Allowed Header Size in the Istio Ingress Gateway

• Increase the maximum allowed request header size in the Istio ingress gateway using an EnvoyFilter.

Note: In environments where Istiod (control plane) and the Istio ingress gateway are deployed in separate namespaces, the EnvoyFilter must be created in the namespace where the ingress gateway workload is deployed.

1. Create an EnvoyFilter similar to the following:
   
   

   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: increase-header-size
     namespace: <istio-gateway-namespace>
   spec:
     workloadSelector:
       labels:
         istio: ingressgateway
     configPatches:
     - applyTo: NETWORK_FILTER
       match:
         context: GATEWAY
         listener:
           filterChain:
             filter:
               name: envoy.filters.network.http_connection_manager
       patch:
         operation: MERGE
         value:
           typed_config:
             "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
             max_request_headers_kb: 96

2. Apply the configuration:
   
            kubectl apply -f envoyfilter.yaml
   
   
3. Restart the ingress gateway deployment

kubectl rollout restart deployment <istio-ingressgateway-deployment> -n <istio-gateway-namespace>

This increases the maximum request header size accepted by the gateway.

Option 2 – Reduce the Size of Client Request Headers

• A more sustainable solution is to reduce the size of request headers sent by clients. This helps avoid proxy limits and improves request efficiency.

• Possible approaches include:

  • Reducing the number or size of cookies stored by the application

  • Avoiding storage of large session data within cookies

  • Minimizing large JWT tokens or authentication headers

  • Removing unnecessary custom headers from client requests

  • Ensuring intermediate proxies are not appending redundant X-Forwarded-* headers

• Optimizing client request headers ensures that requests remain within proxy limits and reduces the likelihood of similar issues occurring in the future.

Additional Information

• Although increasing the header size resolves the immediate issue, it is recommended to review the application behavior to minimize unnecessarily large request headers, particularly large cookies or authentication tokens.
• Reducing header size improves performance and avoids proxy limitations across the environment.