Error: 502 Bad Gateway or 504 Gateway Timeout after migrating to Istio HTTPRoute

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Applications migrated from ingress-nginx to Istio Service Mesh using the Kubernetes Gateway API (HTTPRoute) may experience intermittent HTTP 502 or 504 errors. This article provides steps to determine if the error originates from the Istio Gateway or the backend upstream service.

Symptoms:

• Intermittent HTTP 502 (Bad Gateway) or 504 (Gateway Timeout) reported by clients.
• Istio proxy logs show status codes with the via_upstream flag.
• Issue occurs following a migration from another ingress controller.

Environment

• VMware Cloud Foundation (VCF)
• VMware vSphere Kubernetes Service (TKG)
• Istio Service Mesh
• Kubernetes Gateway API (HTTPRoute)

Cause

The errors are typically returned by the backend application (upstream) rather than the Istio Gateway itself. When Istio logs indicate via_upstream, it signifies the proxy received the error from the backend service and passed it to the client. Misaligned timeout values or backend service crashes during migration often cause these symptoms.

Resolution

1. Identify the Source of the Error: Examine the istio-proxy logs for the gateway pod:

   kubectl logs -l istio=ingressgateway -n #### | grep -E "502|504|500"

   If the log entry contains via_upstream, the backend service is generating the error.

2. Analyze Backend Logs: Correlate the timestamp from the Istio log with the specific backend pod logs identified in the HTTPRoute mapping.

3. Check Timeout Alignment: Ensure that the Istio Gateway timeouts are not shorter than the backend application response times.

4. Verify Topology: Confirm if any external Load Balancers or Proxies sit in front of the Istio Gateway that might be timing out before Istio processes the request.