Performance tuning recommendations for pod to pod Antrea networking in a VKS environment

Products

• VMware NSX
• VMware vSphere Kubernetes Service

Issue/Introduction

This article performs tuning recommendations that may provide higher throughput for Antrea pod to pod networking.

Environment

VMware NSX
Antrea
VMware Kubernetes Services

Resolution

Guest Tuning:

Tuning Configuration	Value	Instruction	Notes
Guest MTU	8900	ip link set eth0 mtu 8900	High performance use cases e.g. storage, AI etc. will generally use jumbo frames.
LRO / GRO	LRO off, GRO on	ethtool -K eth0 lro off gro on	Default configuration
UDP flow hashing	source and destination ports (Antrea overlay UDP header)	ethtool -N eth0 rx-flow-hash udp4 sdfn	This will already be configured due to the vmx setting ethernet0.udpRSS = "1" in the next line Validate it is set with: ethtool -n eth0 rx-flow-hash udp4
VM settings	ethernet0.ctxPerDev = "3" ethernet0.pnicFeatures = "4" ethernet0.udpRSS = "1"	When creating the VKS cluster, edit the Advanced Parameters for the VM class and configure these attributes.	These values cannot be changed once the cluster is created. ethernet0.ctxPerDev = "3" By default, ESXi uses a single execution context (or "world") for the transmit (Tx) path of a virtual NIC. This means one CPU core handles all outgoing packets for that vNIC. Setting this to "3" allows the VMkernel to spawn multiple helper threads to handle the transmit load of a single vNIC. ethernet0.pnicFeatures = "4" The value "4" specifically enables RSS (Receive Side Scaling) support at the pNIC level for the VM. ethernet0.udpRSS = "1" Enables UDP RSS based on UDP ports (not just source and destination IP addresses)

 

ESX Host Tuning:

The ESX host should be prepared for Enhanced Datapath Standard instead of Standard mode, see Enhanced Data Path. Note this is the default configuration for NSX 9.x environments.

Additional Information

Troubleshooting MTU issues and fragmentation in NSX fabric