Mitigating VMXNET3 adapter packet drops due to single receive queue

Products

• VMware vSphere ESXi

Issue/Introduction

• Few Virtual Machines experiences communication problems, including dropped ICMP pings, while other Virtual Machines on the same ESXi host and VLAN are unaffected.
• Migrating the virtual machine to another host does not resolve the issue.
• vsish output reveal incrementing counters for running out of buffers and # of times the 1st ring is full.

cat /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxSummary
stats of a vmxnet3 vNIC rx queue {
   LRO pkts rx ok:0
   LRO bytes rx ok:0
   pkts rx ok:194463553
   bytes rx ok:35272181268
   unicast pkts rx ok:22391322
   unicast bytes rx ok:7656447698
   multicast pkts rx ok:167997988
   multicast bytes rx ok:27242955958
   broadcast pkts rx ok:4074243
   broadcast bytes rx ok:372777612
   running out of buffers:108688339
   pkts receive error:0
   1st ring size:4096
   2nd ring size:32
   # of times the 1st ring is full:0
   # of times the 2nd ring is full:0
   fail to map a rx buffer:0
   request to page in a buffer:0
   # of times rx queue is stopped:0
   failed when copying into the guest buffer:0
   # of pkts dropped due to large hdrs:0
   # of pkts dropped due to max number of SG limits:0
   pkts rx via data ring ok:0
   bytes rx via data ring ok:0
   Whether rx burst queuing is enabled:0
   current backend burst queue length:0
   maximum backend burst queue length so far:0
   aggregate number of times packets are requeued:0
   aggregate number of times packets are dropped by PktAgingList:0
   # of pkts dropped due to large inner (encap) hdrs:0
   number of times packets are dropped by burst queue:0
   number of times packets are dropped by rx try lock queueing:0
   number of packets delivered by burst queue:0
   number of packets dropped by packet steering:0
   number of memory region lookup pass in Rx.:0
   number of packets dropped due to pkt length exceeds vNic mtu:0
   number of packets dropped due to pkt truncation:0
   amount of memory regions mapped for the vnic:0
   total amount of memory regions registered for all vnics on the host:713088
}

• Increasing the VMXNET3 receive (Rx) ring buffers to their maximum values fails to resolve the dropped packets.
• To verify the number of receive queues on VMXNET3 adapter, perform the following steps:
  • Open an SSH session to the ESXi host where the target Virtual Machine is currently running and log in with root privileges.
  • Identify the PortNum and SwitchName using the following command:
    • net-stats -l | grep -i <VM_Name>
  • Using the variables gathered in Step 2, query the rxqueues directory for that specific VMXNET3 port using vsish utility:
    • vsish -e ls /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxqueues/
  • The command will return a list of numbered directories, starting from 0/. Each directory represents an active receive queue on that VMXNET3 adapter:

[root@<hostname>:~] vsish -e ls /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxqueues/
0/

Environment

VMware vSphere ESXi

Cause

• The Virtual Machine(s) is constrained by a single receive (Rx) queue on the VMXNET3 adapter. 
• This configuration forces all incoming network interrupts to be processed by a single vCPU.
• Under heavy network load, this single vCPU becomes a processing bottleneck. Because the vCPU cannot drain the network queue fast enough, the ring buffers exhaust their capacity (even at maximum size), resulting in dropped packets.

Resolution

To increase the number of receive (RX) queues for your VMXNET3 adapter, you need to configure Receive Side Scaling (RSS) within the Guest Operating System.

The maximum number of RX queues a VM can use is bounded by the number of vCPUs assigned to that VM.

For example, a VM with 4 vCPUs can have a maximum of 4 RX queues.

Increasing queues for a VMXNET3 adapter in Linux:

• To increase the number of queues, run the following commands:
• modprobe vmxnet3 num_rqs=<value>
• modprobe vmxnet3 num_tqs=<value>
• Increasing queues for a VMXNET3 adapter in Windows:
• Open Device Manager and expand Network Adapters.
• Right click the VMXNET3 adapter and select Properties.
• Click the Advanced Tab.
• Change the Value for Max Tx queues and Maximum Number of RSS queues.

NOTE:

The value for above parameters cannot exceed the value of vCPUs assigned to the Virtual Machine.

Perform the above steps in a maintenance window and from Web Console of the Virtual Machine.

After performing the above steps, number of queues can be confirmed using the steps mentioned above in above Issue/Introduction section. For example, when the receive queues are set to 4:

[root@esx-05:~] vsish -e ls /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxqueues/
0/
1/
2/
3/