VirtualService sends additional HTTP 400 or 503 error response for previous POST requests' payload

Products

• VMware Avi Load Balancer

Issue/Introduction

This issue is observed on upload applications, and the HTTP errors are seen on the client if:

• The server responds with a 401.
• We forward the 401 to the client.
• But the client has already sent part of the POST payload before processing the 401.
• We consider this payload as a part of a new request and respond with a 400.
• If "Proxy Expect Header" is disabled in the application profile, the client will see a 503 error code instead.

Environment

• Avi version <=30.2.6
• This affects environments with L7 VS flows where client uploads data to servers and the application profile has connection multiplexing enabled.

 

 

Cause

• When connection multiplexing is enabled, the frontend and backend connections are disjoint. 
• When a server sends a error(3xx/4xx) code in the middle of the client uploading the request body, it is proxied back to the client. 
• But because of multiplexing, the frontend connection is not immediately closed.
• Because of this, when the client uploads more data on the same connection, it is treated as a new request and it fails.
• With multiplexing disabled, the frontend and backend connections are mapped 1:1. Thus, when the initial response is sent to the client, the connection is immediately torn down, preventing any additional HTTP error responses from Avi.

Resolution

• A temporary workaround for this would be to disable Connection Multiplexing in the Application Profile associated to the VS.
• Refer to this doc for more information on Connection Multiplexing.
• To disable this navigate to Templates>Profiles>VS Application Profile and uncheck the Connection Multiplexing checkbox.
• Please upgrade to 30.2.7 where this issue is fixed.
  • Please find the bug details in the Release notes: AV-267951
• The behavior starting with 30.2.7 will be:
  • When a non-2xx response is received from server while the client request body is still in transit, AVI now explicitly sets "Connection: close" and discards any remaining upload data so the connection can be released cleanly.
  • This resolves the 400 Bad Request errors that customers were previously seeing when the connection was left in an inconsistent state.
  • When Proxy Expect Header is enabled, AVI now holds only 2xx responses until the full client request body has been received. Non-2xx responses (3xx and above) are forwarded to the client immediately without being held.