Troubleshooting NIC errors and other network traffic faults in ESXi
Article ID: 341594
Updated On: 04-30-2025
Products
Issue/Introduction
The esxcli network nic stats get -n vmnic# command may show error or drop counters that are greater than zero, on one or more physical network adapters.
Below is an example output:$ esxcli network nic stats get -n vmnic2
NIC statistics for vmnic2 Packets received: 701280499176 Packets sent: 687061948450 Bytes received: 664124780523852 Bytes sent: 676938646792793 Receive packets dropped: 2452783244 Transmit packets dropped: 0 Multicast packets received: 976222150 Broadcast packets received: 0 Multicast packets sent: 0 Broadcast packets sent: 0 Total receive errors: 0 Receive length errors: 0 Receive over errors: 0 Receive CRC errors: 0 Receive frame errors: 0 Receive FIFO errors: 0 Receive missed errors: 0 Total transmit errors: 0 Transmit aborted errors: 0 Transmit carrier errors: 0 Transmit FIFO errors: 0 Transmit heartbeat errors: 0 Transmit window errors: 0
Cause
The data these counters display are additive of issues that are external to the ESXi kernel and only what is being reported to the ESXi host from the NIC driver.
When these counters show higher values, the rate that the numbers increase can be a helpful tool. To monitor the counters and see if the numbers are increasing, use the command below:
$ watch esxcli network nic stats get -n vmnicX
It is important to note that in some instances the numbers associated with the counters can be rather large or it may not be known if the increased values are new or from a previous issue. It may be helpful to clear them before proceeding with the investigation. To clear the values the ESXi host will need to be rebooted:
- Record the current values of the counters.
- Place the ESXi host into Maintenance Mode.
- Perform a reboot.
- Once the ESXi host is back up the counters values will be reset to zero.
In a healthy environment, "errors" should either be zero, or very small as a percentage of the overall total.
Common Counter Meanings:
- Receive packets dropped:
- This counter is an advisement of packets being dropped prior to the packet reaching the ESXi kernel. Quite often this counter is a combination of the values from other counters that can be found in the Private Statistics section of the
nicinfo.sh.txtfile that is contained in thecommandsdirectory of ESXi host log bundles. - Example:
- This counter is an advisement of packets being dropped prior to the packet reaching the ESXi kernel. Quite often this counter is a combination of the values from other counters that can be found in the Private Statistics section of the
NIC statistics for vmnic0: Packets received: 1880190132 Packets sent: 1887598404 Bytes received: 264206918890 Bytes sent: 269243305508 Receive packets dropped: 11592374
-
- From the basic output above it is shown that there is a high record of
Receive packets dropped. - Specifically within the Private Statistics for vmnic0 the following received (
rx) discards are noted:
- From the basic output above it is shown that there is a high record of
[rxq1] discards rx: 11585755
[rxq2] discards rx: 6619
-
- By adding the discard values of 11585755 and 6619 together the
Receive packets droppedvalue is 11592374.
- By adding the discard values of 11585755 and 6619 together the
- Receive length errors:
- This counter is an indication that there are issues with the actual size of the packet versus what the size of the packet being reported via the packet header.
- Example:
- A packets actual size is 80 bytes long.
- The packet header advises the size of the packet is 56 bytes long.
- The output of the NIC stats shows:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 200Receive length errors: 200
- Receive missed errors:
- Indicates issues with the NIC being able to store or process packets due to hardware limitations.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 69588Receive length errors: 0Receive over errors: 0Receive CRC errors: 1Receive frame errors: 0Receive FIFO errors: 0Receive missed errors: 69587
- Receive over errors:
- The packets that are discarded by the hardware buffer of the card.
- The total receive errors are any CRC errors as well as possibly values from the other counters.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 9464Receive length errors: 0Receive over errors: 9463Receive CRC errors: 1Receive frame errors: 0Receive FIFO errors: 0Receive missed errors: 0Total transmit errors: 0Transmit aborted errors: 0Transmit carrier errors: 0Transmit FIFO errors: 0Transmit heartbeat errors: 0Transmit window errors: 0
- The packets that are discarded by the hardware buffer of the card.
- FIFO or Missed errors:
- These counters will increment and accumulate if physical NIC is not able to handle the peak load of incoming packets with current rx ring buffer size.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 1335389Receive length errors: 0Receive over errors: 9463Receive CRC errors: 1Receive frame errors: 0Receive FIFO errors: 1325896Receive missed errors: 29Total transmit errors: 0Transmit aborted errors: 0Transmit carrier errors: 0Transmit FIFO errors: 0Transmit heartbeat errors: 0Transmit window errors: 0
- CRC:
- The CRC stands for "Cyclic Redundancy Check". The FCS (Frame Check Sequence) field contains a 4-byte CRC value used for error checking. When a source host assembles a packet, it performs a CRC calculation on all fields in the packet except the Preamble, SFD (Start Frame Delimiter), and FCS using a predetermined algorithm. The source host stores the value in the FCS field and transmits it as part of the packet. When the packet is received by the destination host, it performs a CRC test again by using the same algorithm. If the CRC value calculated at the destination host does not match the value in the FCS field, the destination host discards the packet, considering this as a CRC Error.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 1Receive length errors: 0Receive over errors: 0Receive CRC errors: 1
Resolution
If any NICs show errors, consult with the hardware vendor to troubleshoot the physical NIC errors as they are only reported to the ESXi hosts.
Below are further troubleshooting steps that may help while investigating physical NIC errors:
- Check that the driver/firmware of the vmnic is up to date.
- To check the driver, follow Determining Network/Storage firmware and driver version in ESXi (323110).
- For more information on drivers and firmware, see FAQ: Recommendation for Driver/Firmware (318542)
- It is possible to mitigate FIFO or Missed errors by increasing the Rx buffer ring size on the physical NIC.
- Increasing the Rx and Tx values on the hardware side and within the guest OS can significantly enhance VM performance, especially in high I/O environments. However, the effectiveness of this adjustment depends on factors such as the application, operating system, and hardware in use. Therefore, the recommendations should come from the respective application, OS, and hardware vendors to ensure compatibility and optimal performance.
- Use the following commands to check the maximum (preset) ring buffer size and current ring buffer size.
$ esxcli network nic ring preset get -n vmnicX $ esxcli network nic ring current get -n vmnicX
Information Regarding Physical NIC Ring Buffers:
A ring buffer is used to allocate a section of memory which is like a temporary holding area for packets whose packet rate may be so high that the code required to process them has trouble keeping up.
Each type of network adapter has a preset maximum which is determined by the device driver. This is revealed by the above command featuring preset.
The preset maximum is usually higher than the default setting which is allocated when ESXi is installed. This is revealed by the above command featuring current.
Below is an example of the output from the buffer commands above (the output may differ between vendors):
Current Ring Size: RingInfo: RX: 1023 RX Mini: 0 RX Jumbo: 0 TX: 1023
Preset Maximum Ring Size: RingInfo: Max RX: 4095 Max RX Mini: 0 Max RX Jumbo: 0 Max TX: 4095
With the development of higher and higher NIC speeds and physical switches, and applications which involve higher volumes of TCP/IP packets and packet rates, it is becoming more common to see situations where increasing the physical uplink ring buffer sizes to the preset maximums, will reduce the possibility of drivers being unable to cope with high packet arrival rates, however confirmation from the NIC vendor is recommended prior to making any changes.
Command to increase both rx and tx: "esxcli network nic ring current set -n vmnic4 -r 4096 -t 4096"
for receive ring buffer only: "esxcli network nic ring current set -n vmnic4 -r 4096"
Max value will depend on the one got from the get command "esxcli network nic ring preset get -n vmnicX"
NOTE : Confirm with hardware vendor the recommended value to be set
Additional Information
There may be a log message that populates for vSAN environments stating "High pNic error rate detected" which takes data from a specific group of the counters listed above that can impact vSAN performance. Please refer to Alarm about high pNIC error rate being detected for more information if this error message is present.
Feedback
