Mitigating VMXNET3 adapter packet drops due to single receive queue

Products

• VMware vSphere ESXi

Issue/Introduction

• Few Virtual Machines experiences communication problems, including dropped ICMP pings, while other Virtual Machines on the same ESXi host and VLAN are unaffected.
• Migrating the virtual machine to another host does not resolve the issue.
• vsish output reveal incrementing counters for running out of buffers and # of times the 1st ring is full.

cat /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxSummary
stats of a vmxnet3 vNIC rx queue {
   LRO pkts rx ok:0
   LRO bytes rx ok:0
   pkts rx ok:194463553
   bytes rx ok:35272181268
   unicast pkts rx ok:22391322
   unicast bytes rx ok:7656447698
   multicast pkts rx ok:167997988
   multicast bytes rx ok:27242955958
   broadcast pkts rx ok:4074243
   broadcast bytes rx ok:372777612
   running out of buffers:108688339
   pkts receive error:0
   1st ring size:4096
   2nd ring size:32
   # of times the 1st ring is full:0
   # of times the 2nd ring is full:0
   fail to map a rx buffer:0
   request to page in a buffer:0
   # of times rx queue is stopped:0
   failed when copying into the guest buffer:0
   # of pkts dropped due to large hdrs:0
   # of pkts dropped due to max number of SG limits:0
   pkts rx via data ring ok:0
   bytes rx via data ring ok:0
   Whether rx burst queuing is enabled:0
   current backend burst queue length:0
   maximum backend burst queue length so far:0
   aggregate number of times packets are requeued:0
   aggregate number of times packets are dropped by PktAgingList:0
   # of pkts dropped due to large inner (encap) hdrs:0
   number of times packets are dropped by burst queue:0
   number of times packets are dropped by rx try lock queueing:0
   number of packets delivered by burst queue:0
   number of packets dropped by packet steering:0
   number of memory region lookup pass in Rx.:0
   number of packets dropped due to pkt length exceeds vNic mtu:0
   number of packets dropped due to pkt truncation:0
   amount of memory regions mapped for the vnic:0
   total amount of memory regions registered for all vnics on the host:713088
}

• Increasing the VMXNET3 receive (Rx) ring buffers to their maximum values fails to resolve the dropped packets.
• To verify the number of receive queues on VMXNET3 adapter, perform the following steps:
  • Open an SSH session to the ESXi host where the target Virtual Machine is currently running and log in with root privileges.
  • Identify the PortNum and SwitchName using the following command:
    • net-stats -l | grep -i <VM_Name>
  • Using the variables gathered in Step 2, query the rxqueues directory for that specific VMXNET3 port using vsish utility:
    • vsish -e ls /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxqueues/
  • The command will return a list of numbered directories, starting from 0/. Each directory represents an active receive queue on that VMXNET3 adapter:

[root@<hostname>:~] vsish -e ls /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxqueues/
0/

Environment

VMware vSphere ESXi

Cause

• The Virtual Machine(s) is constrained by a single receive (Rx) queue on the VMXNET3 adapter. 
• This configuration forces all incoming network interrupts to be processed by a single vCPU.
• Under heavy network load, this single vCPU becomes a processing bottleneck. Because the vCPU cannot drain the network queue fast enough, the ring buffers exhaust their capacity (even at maximum size), resulting in dropped packets.

Resolution

• To increase the number of receive (RX) queues for your VMXNET3 adapter, you need to configure Receive Side Scaling (RSS) within the Guest Operating System.
• The maximum number of RX queues a VM can use is bounded by the number of vCPUs assigned to that VM.
  • For example, a VM with 4 vCPUs can have a maximum of 4 RX queues.

Increasing queues for a VMXNET3 adapter in Linux:

• To increase the number of queues, run the following commands:
  • modprobe vmxnet3 num_rqs=<value>
  • modprobe vmxnet3 num_tqs=<value>

Increasing queues for a VMXNET3 adapter in Windows:

• Open Device Manager and expand Network Adapters.
• Right click the VMXNET3 adapter and select Properties.
• Click the Advanced Tab.
• Change the Value for Max Tx queues and Maximum Number of RSS queues.

**NOTE:

1. The value for above parameters cannot exceed the value of vCPUs assigned to the Virtual Machine.
2. Perform the above steps in a maintenance window and from Web Console of the Virtual Machine.

After performing the above steps, number of queues can be confirmed using the steps mentioned above in above Issue/Introduction section. For example, when the receive queues are set to 4:

[root@esx-05:~] vsish -e ls /net/portsets/<SwitchName>/ports/<PortNum>/vmxnet3/rxqueues/
0/
1/
2/
3/