File upload is slow through AVI Load balancer Virtual service
Article ID: 428007
Updated On:
Products
Issue/Introduction
The customer reported performance degradation while uploading a large file or multiple files concurrently through the AVI Load Balancer.
When the same upload operation was performed by bypassing AVI and connecting directly to the backend server, the transfer completed successfully without any noticeable delay.
Environment
Avi Load Balancer : 22.1.x , Version below 30.2.2 & 31.1.1
Virtual Service: L7 / L4 with TCP Proxy
Cause
Collect the Virtual service packet capture and observe if we are noticing the following behaviour
Collect-Virtual-service-packet-capture
- Avi Load Balancer implements TCP delayed acknowledgment (delayed ACK) to reduce the number of ACK packets sent. With delayed ACK enabled, Avi may acknowledge every second received TCP segment or send an ACK after a timeout (typically up to ~100 ms) if no additional segment arrives.
- During a file upload from the client to the server, at the initial stage of data transfer, the Avi Load Balancer acknowledges for every packet as the TCP connection ramps up.
- As the data transfer stabilises, Avi gradually applies the delayed ACK mechanism to reduce ACK traffic. In this phase, Avi sends one ACK for every two TCP segments received from the client, which is expected TCP behavior when delayed ACK is in effect.
- Over time, if the client is unable to send multiple TCP segments back-to-back (for example, due to congestion or window limitations), TCP retransmissions may occur. In Below screenshot source machine is re-transmitting the packets.
- Each retransmission triggers an immediate ACK from Avi. As retransmissions increase, the client’s effective send rate decreases, which can result in slower file upload performance.
Resolution
If traffic captures exhibit patterns similar to those described above, the behavior is likely caused by the Delayed ACK mechanism implemented in the AVI Load Balancer.
To mitigate this behaviour, the Delayed ACK configuration on the AVI Load Balancer can be adjusted. This can be achieved by either:
- Reducing the Delayed ACK timer
- Disabling the Delayed ACK mechanism.
Support for modifying the Delayed ACK timer was introduced in the following AVI versions:
- 30.2.2
- 31.1.2
Support for disabling Delayed ACK and modifying Delayed ACK timer is introduced in the following AVI versions:
- 31.2.1
>> Steps to reduce the delayed_ack timers in 30.2.2/31.1.2 version
Step-1: Create a new network profile
Navigate to Templates >> Profiles >> Network click on CREATE button
Please select the below settings and leave remaining settings as it is.
Step-2: Login to the shell of Leader controller node and run the below commands
[admin:##-##-##-##]: > configure networkprofile <Network-profile-name>
[admin:##-##-##-##]: networkprofile> profile tcp_proxy_profile
[admin:##-##-##-##]: networkprofile:profile:tcp_proxy_profile> delayed_ack_timer_delay <value less than 100 milli seconds>
[admin:##-##-##-##]: networkprofile:profile:tcp_proxy_profile> save
[admin:##-##-##-##]: networkprofile> saveSteps to disable the delayed_ack setting from 31.2.1/30.2.6 version
Step-1: Create the new network profile same as above
Step-2: Login to the shell of Leader controller node and run the below commands
[admin:##-##-##-##]: > configure networkprofile <Network-profile name>
[admin:##-##-##-##]: networkprofile> profile tcp_proxy_profile
[admin:##-##-##-##]: networkprofile:profile:tcp_proxy_profile> no delayed_ack_mode
[admin:##-##-##-##]: networkprofile:profile:tcp_proxy_profile> save
[admin:##-##-##-##]: networkprofile> save
Feedback
