File upload requests exceeding 16MB are timing out with an HTTP 408 Request Timeout error.

Products

• VMware Avi Load Balancer

Issue/Introduction

• File uploads >16MB consistently fail.

• The upload process stalls and times out after ~30 seconds.

• Packet capture shows multiple Zero Window advertisements sent by the SE (VIP) on the frontend/client side, leading to POST request cancellation.
  
  
  

Cause

• The backend server advertises a small TCP receive window (65KB), which gets fully utilized as the SE sends data. Due to the high round-trip time (~110ms), ACKs from the server are delayed, preventing the SE from forwarding more data. As a result, the SE's internal receive buffer saturates, causing it to advertise a Zero Window to the client.
• This flow control bottleneck leads to a stalled client POST request, eventually resulting in an HTTP 408 (Request Timeout) error.

Resolution

 

• The server advertises a maximum receive window of 65KB.

• The SE sends 65KB of data, fills the server’s window, and must then pause client-side receive until the server ACKs more data.

• Due to a high RTT (~110ms) between the SE and the server, ACKs from the server are delayed.

• This causes backpressure on the client side, eventually leading to client-side receive buffer saturation and Zero Window advertisements from the SE.

• The POST is stalled until ACKs arrive from the server — which happens slowly — resulting in upload timeout and HTTP 408 error.

The server advertises a 65K receive window, which is fully utilized (RED), causing the SE to pause client-side receive. Upon receiving ACK in pkt 43804, the server window opens and the client window updates (BLUE), resuming flow. The stall persists until the server ACKs and reopens the window (GREEN).

 

 

Suggestions:

1. Static Receive window sizing (e.g., ~1MB) may help, but beware of:

   • Increased memory pressure on SE.
     
     

2. Increase the server's receive window size:

   • If applicable, enable Window Scaling.

3. Reduce delayed ACK behavior on the server:

   • Some servers delay ACKs intentionally (e.g., Nagle’s Algorithm + delayed ACK) — tune TCP settings to reduce this.