"pkts receive error" observed on vmxnet3 stats

Products

• VMware vSphere ESXi

Issue/Introduction

Follow the steps below to get the VMXNET3 stats:

• Run the command "net-stats -l" in an SSH session to the ESXi host managing the VM that has the issue:

• From the output of the above command, get the PortNum and SwitchName values that correlate with your VM, and substitute their values in the below command:

vsish -e get /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxSummary

 

Example:

Environment

VMware vSphere ESXI 7.x
VMware vSphere ESXi 8.x

Cause

When multiple NetPoll worlds are able to deliver packets to the same vNIC, the wait time increases, resulting in higher latency.
In version 7.0, packets are queued initially and retried in the next attempt. This can cause contention either at the vNIC level or the queue level, which may lead to the dropping of other incoming packets. As a result, we observe an increase in the "packets receive error" counter.

Resolution

Currently, there is no resolution.

Workaround:

The two workarounds below may help to resolve the issue and can be tried out, but these are just workarounds; the issue could be because of other reasons as well.

• Workaround 1:  
  • Vmxnet3RxQueueLimit:
    • To workaround this issue, there is a config option to increase the queueLimit.
    • Vmxnet3RxQueueLimit - maximum number of packets to queue in the backend if unable to hold the lock.
    • Default value of Vmxnet3RxQueueLimit is 1024 and maximum value is 4096
    • NOTE: before making changes of Vmxnet3RxQueueLimit, place ESXi host into Maintenance Mode.

• • To get the value:

    • esxcfg-advcfg -g /Net/Vmxnet3RxQueueLimit

  • To set the value:

    • esxcfg-advcfg -s 2048 /Net/Vmxnet3RxQueueLimit

• Workaround 2: Note: Nic disable/enable or VM reboot will be required to get this in effect.

• • EthernetX.pnicFeatures = "4":
    
    • When multiple vmnic are trying to deliver packets to the same vnic, this will lead to contention and hence a drop of packets.
    • ethernetX.pnicFeatures = "4" This setting will avoid the contention at vnic level.
    • To add this key-value pair, we need to edit the concerned VM VMX file. Make sure VM is powered off before adding this setting.
    • Add ethernetX.pNicFeatures = "4" , ("X" here is the number of virtual network cards to which the feature should be added)
  • Find the below steps to add the entry in VMX file/VM settings:
    
    • For vSphere 8.0: Set Advanced Virtual Machine Attributes 
    • For vSphere 7.0: Set Advanced Virtual Machine Attributes