The output of esxtop shows dropped receive packets at the virtual switch
Article ID: 312565
Updated On:
Products
VMware vSphere ESXi
VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
Issue/Introduction
The network screen output in esxtop show dropped receive packets (%DRPRX) at the virtual switch port.
Warning: The changes described in this article may impact guest networking at the time of the change. You may need to reboot the guest to recover. VMware recommends you to schedule downtime for applications running within the guest to which you are making changes prior to performing the steps.
Warning: The changes described in this article may impact guest networking at the time of the change. You may need to reboot the guest to recover. VMware recommends you to schedule downtime for applications running within the guest to which you are making changes prior to performing the steps.
Environment
VMware vSphere ESXi
Cause
Virtual machine operating systems allocate memory (RAM) to process network traffic. Under heavy load, the guest OS may not have allocated sufficient memory to store incoming network packets.
Resolution
Esxtop might show receive packets dropped at the virtual switch. If the virtual machine’s network driver runs out of receive (RX) buffer memory the packets are treated in a FIFO (first in first out basis) which can result in degraded performance. Even though esxtop shows the packets as dropped at the virtual switch, they are actually dropped between the virtual switch and the guest operating system driver.
The number of dropped packets can be reduced by increasing the Rx buffers for the virtual network driver.
Warning: The changes described in this article may impact guest networking at the time of the change. You may need to reboot the guest to recover. VMware recommends you to schedule downtime for the applications running within the guest to which you are making changes prior to performing the steps.
VMXNET3
For Linux guest operating systems
The ring size setting can be modified from within the guest operating system.
You can view the current values using the command:
ethtool -g <interface>
where <interface> is the interface name as it appears in your OS, e.g. eth0.
You can set a value using a capital G, followed by the interface name, followed by pairs of settings and values, for example:
ethtool -G <interface> rx 4096
ethtool -G <interface> rx 4096 rx-jumbo 4096 rx-mini 2048 tx 4096
The rx setting refers to receive ring buffer #1, which is used for receiving packets other than jumbo frames (up to 1500 bytes). The rx-jumbo setting refers to ring buffer #2, which is used for jumbo frames. The tx and rx-mini settings are for transmit and tiny frames.
A Linux virtual machine enabled with Large Receive Offload (LRO) functionality on a VMXNET3 device might experience packet drops on the receiver side when the Rx ring #2 runs out of memory. This occurs when the virtual machine is handling packets generated by LRO. Up to ESXi 5.5 Update 3, the maximum ring size for this parameter is 2048. For ESXi 5.5 Update 3 Patch 08, and ESXi 6.0 Patch 3 and later, the maximum value is 4096.
Important Notes:
- Changes made using ethtool will generally not persist a reboot. To keep these changes after a restart, you will need to save these changes using a method such as a startup script (e.g. ETHTOOL_OPTS in /etc/sysconfig/network-scripts/ifcfg-eth0) or a configuration manager (e.g. nmcli connection modify ... ethtool.ring-rx). The method varies by Linux distribution. Contact your OS vendor for information on how to do this.
- Even when ESXi supports a higher maximum value, the actual usable maximum value may be limited by the Linux guest operating system's kernel version and other variables.
For Windows guest operating systems
In ESXi/ESX 4.0 Update 2 and later, you can configure the Rx Ring #1 Size, Rx Ring #2 Size, Tx Ring Size, Small Rx Buffers, and Large Rx Buffers parameters in the Device Properties from the Device Manager in Windows guest operating systems. These settings cannot be adjusted in the Windows guest operating system in ESXi/ESX 3.x.x and earlier.
The default value of the size of the first Rx ring, Rx Ring #1 Size, is 512. The maximum is 4096.
When jumbo frames are enabled, the network card also uses a second ring, Rx Ring #2 Size. The default value of RX Ring #2 Size is 4096 and the maximum is 8192.
You can also modify the number of small Rx buffers separately using the Small Rx Buffers parameter. The maximum value is 8192.
You can also modify the number of small Rx buffers separately using the Small Rx Buffers parameter. The maximum value is 8192.
For some processes (for example, traffic that arrives in burst), you might need to increase the size of the ring, while for others (for example, applications that are slow in processing receive traffic) you might increase the number of receive buffers. The number of large buffers that are used in both RX Ring #1 and #2 sizes when jumbo frames are enabled is controlled by Large Rx Buffers. The default value of Large Rx Buffers is 768.
For Solaris guest operating systems
- If the driver is already loaded, unload it by entering the following command:
# rem_drv vmxnet3s
- To change the size of Rx rings, edit the vmxnet3.conf file available in the /kernel/drv/amd64 and /kernel/drv/ directories.
- Reload the driver by entering the following commands:
# add_drv -i "pciex15ad,7b0" vmxnet3s
# ifconfig vmxnet3sx plumb
E1000 Virtual Network Driver
Linux:
For the E1000 virtual network driver in a Linux guest operating system, Rx buffers can be modified from the guest operating system in exactly the same way as on the physical machine. The default value is 256, and the maximum value that can be manually configured is 4096. Determine an appropriate setting by experimenting with different buffer sizes.
To determine the appropriate setting by running the ethtool -G ethX rx value command.
Windows:
For the Intel pro driver in Windows, receive buffers can be modified from the guest operating system in exactly the same way as on the physical machine. The default value may vary depending on the driver used. To determine the appropriate setting by experimenting with different buffer size, load the Intel pro driver to the guest operating system and modify the Receive Buffers in the driver’s property.
VMXNET2 / Enhanced VMXNET2 Virtual Network Driver
To increase the default value, add this line in the virtual machine’s .vmx configuration file:
Ethernet x.numRecvBuffers=128
Where x refers to your virtual NIC.
On ESX 4.x, the maximum is 512 and the default is 150. On ESXi 3.x, the maximum Rx buffers supported are 128 and the default is 120.
Additional Information
The esxtop utility is also very useful to determine which physical uplinks (vmnics) are carrying the traffic for which workloads.
For example, if you SSH into the ESXi host with root privileges, you can enter:
esxtop
Then when the display appears, enter the lower case "n" (for network)
You will see the screen change to display line items for each workload and vmnic, and which vmnic is being used to carry the traffic.
Generally the value in the "Pnic" column will show like vmnic0, vmnic1, etc.
It may show "All(2)" or "All(1)" or "All(4)" for example. In this case, one or more uplinks are configured in an etherchannel (also referred to as a portchannel), either as a simple portchannel or as part of an LACP configuration.
Feedback
Yes
No
Powered by
