Packet drops observed on vmxnet3 when packet length greater than vNIC's MTU

Products

• VMware vSphere ESXi

Issue/Introduction

Packet drops increase when vmxnet3 is used on vsphere67u2 or later.

To get vmxnet3 stats, run the following command in an SSH session to the ESXi host managing the VM in question:

net-stats -l

From the output of the above command, note the PortNum and SwitchName values that correlate with your VM, and substitute their values in the below command:

vsish -e cat /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxSummary

Example:

vsish -e cat /net/portsets/DvsPortset-0/ports/67108902/vmxnet3/rxSummary
stats of a vmxnet3 vNIC rx queue {
...
      number of packets dropped due to pkt length exceeds vNic mtu:462727 <=== this counter increases
...

Environment

VMware vSphere

VMware ESXi

Cause

vSphere 6.7 Update 2, MTU check is now enforced on packets on receive path to not allow non-LRO (Large Receive Offload) packets greater than vNIC MTU.

This limit was not enforced in previous releases.

Resolution

To resolve this issue, config options are provided in vSphere 6.7 P01 at two different granularities.
Note: The default will still be to enforce a stricter MTU check.

Use one of these options to re-enable the previous behavior (that is, to avoid the MTU check):

• Host-wide config option:

esxcli system settings advanced set -o "/Net/Vmxnet3NonTsoPacketGtMtuAllowed" -i 1
Note: This configuration is applicable over all vmxnet3 vNICs (host wide). This setting will be then applied to each VM that will be powered on after making this change

OR

• vNIC specific config option:

Use ethernet0.rxAllowPktGtMtu = "1" in the .vmx file:

Where, "ethernet0" should be replaced by the specific vNIC on which the configuration is to be applied.
Please use the VMware KB article below to follow steps on how to:

Modifying advanced virtual machine settings using vSphere Client 

Workaround:
To work around this issue on vSphere 6.7 Update 2 and later versions, change the receiving vNIC's MTU (inside the VM) as per the packet lengths expected.

Example:
If vSwitch MTU is 9000, then packets up to 9000 bytes in length can be delivered as non-LRO packets. In this case, increase the vNIC's MTU to 9000.