On Vsphere ESXi 8.0 TCP connections not able to offload on Nvidia Bluefield 2 SmartNIC

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

TCP throughput degradation on SmartNIC setup when running on Nvidia Bluefield 2 SmartNIC vs. performance NIC.

Environment

VMware vSphere ESXi 8.0

Cause

The maximum window scale supported by BF2 SmartNIC is 7. When a larger scale is used by the guest, TCP connections won't be able to offload onto hardware, hence cause performance degradation.

Resolution

Currently there is no resolution to the issue. The issue will be resolved in future releases.
 

Workaround:

To workaround the issue, please follow the below mentioned instruction:

On Linux virtual machine, reduce the wmem_max and rmem_max:

echo 'net.core.wmem_max=4194304' >> /etc/sysctl.conf
echo 'net.core.rmem_max=4194304' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 87380 4194304' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 87380 4194304' >> /etc/sysctl.conf
sysctl -p