Troubleshooting NIC errors and other network traffic faults in ESXi
Article ID: 341594
Updated On:
Products
Issue/Introduction
To check the details of the physical network adapters/vmnic from the ESXi host, follow the steps below
- Login to the ESXi host as user root via putty/ssh
- Run the command
esxcli network nic listoresxcfg-nics -l
Sample output of the command :-
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------- ------------ ------ ------------ ----------- ------ ------ ----------------- ---- -----------
vmnic0 0000:01:00.0 ixgben Up Up 1000 Full ec:f4:bb:##:##:## 1500 Intel(R) Ethernet Controller X540-AT2
vmnic1 0000:01:00.1 ixgben Up Up 1000 Full ec:f4:bb:##:##:## 1500 Intel(R) Ethernet Controller X540-AT2
The esxcli network nic stats get -n vmnic# command may show error or drop counters that are greater than zero, on one or more physical network adapters.
Below is an example output:$ esxcli network nic stats get -n vmnic2
NIC statistics for vmnic2 Packets received: 701280499176 Packets sent: 687061948450 Bytes received: 664124780523852 Bytes sent: 676938646792793 Receive packets dropped: 2452783244 Transmit packets dropped: 0 Multicast packets received: 976222150 Broadcast packets received: 0 Multicast packets sent: 0 Broadcast packets sent: 0 Total receive errors: 0 Receive length errors: 0 Receive over errors: 0 Receive CRC errors: 0 Receive frame errors: 0 Receive FIFO errors: 0 Receive missed errors: 0 Total transmit errors: 0 Transmit aborted errors: 0 Transmit carrier errors: 0 Transmit FIFO errors: 0 Transmit heartbeat errors: 0 Transmit window errors: 0
Environment
VMware ESXi 7x
VMware ESXI 8x
Cause
The data these counters display are additive of issues that are external to the ESXi kernel and only what is being reported to the ESXi host from the NIC driver.
When these counters show higher values, the rate that the numbers increase can be a helpful tool. To monitor the counters and see if the numbers are increasing, use the command below:
$ watch esxcli network nic stats get -n vmnic#
It is important to note that in some instances the numbers associated with the counters can be rather large or it may not be known if the increased values are new or from a previous issue. It may be helpful to clear them before proceeding with the investigation. To clear the values the ESXi host will need to be rebooted:
- Record the current values of the counters.
- Place the ESXi host into Maintenance Mode.
- Perform a reboot.
- Once the ESXi host is back up the counters values will be reset to zero.
In a healthy environment, "errors" should either be zero, or very small as a percentage of the overall total.
Common Counter Meanings:
- Receive packets dropped:
- This counter is an advisement of packets being dropped prior to the packet reaching the ESXi kernel. Quite often this counter is a combination of the values from other counters that can be found in the Private Statistics section of the
nicinfo.sh.txtfile that is contained in thecommandsdirectory of ESXi host log bundles. - Example:
- This counter is an advisement of packets being dropped prior to the packet reaching the ESXi kernel. Quite often this counter is a combination of the values from other counters that can be found in the Private Statistics section of the
NIC statistics for vmnic0: Packets received: 1880190132 Packets sent: 1887598404 Bytes received: 264206918890 Bytes sent: 269243305508 Receive packets dropped: 11592374
-
- From the basic output above it is shown that there is a high record of
Receive packets dropped. - Specifically within the Private Statistics for vmnic0 the following received (
rx) discards are noted:
- From the basic output above it is shown that there is a high record of
[rxq1] discards rx: 11585755
[rxq2] discards rx: 6619
-
- By adding the discard values of 11585755 and 6619 together the
Receive packets droppedvalue is 11592374.
- By adding the discard values of 11585755 and 6619 together the
- Receive length errors:
- This counter is an indication that there are issues with the actual size of the packet versus what the size of the packet being reported via the packet header.
- Example:
- A packets actual size is 80 bytes long.
- The packet header advises the size of the packet is 56 bytes long.
- The output of the NIC stats shows:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 200Receive length errors: 200
- Receive missed errors:
- Indicates issues with the NIC being able to store or process packets due to hardware limitations.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 69588Receive length errors: 0Receive over errors: 0Receive CRC errors: 1Receive frame errors: 0Receive FIFO errors: 0Receive missed errors: 69587
- Receive over errors:
- The packets that are discarded by the hardware buffer of the card.
- The total receive errors are any CRC errors as well as possibly values from the other counters.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 9464Receive length errors: 0Receive over errors: 9463Receive CRC errors: 1Receive frame errors: 0Receive FIFO errors: 0Receive missed errors: 0Total transmit errors: 0Transmit aborted errors: 0Transmit carrier errors: 0Transmit FIFO errors: 0Transmit heartbeat errors: 0Transmit window errors: 0
- The packets that are discarded by the hardware buffer of the card.
- FIFO or Missed errors:
- These counters will increment and accumulate if physical NIC is not able to handle the peak load of incoming packets with current rx ring buffer size.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 1335389Receive length errors: 0Receive over errors: 9463Receive CRC errors: 1Receive frame errors: 0Receive FIFO errors: 1325896Receive missed errors: 29Total transmit errors: 0Transmit aborted errors: 0Transmit carrier errors: 0Transmit FIFO errors: 0Transmit heartbeat errors: 0Transmit window errors: 0
- CRC:
- The CRC stands for "Cyclic Redundancy Check". The FCS (Frame Check Sequence) field contains a 4-byte CRC value used for error checking. When a source host assembles a packet, it performs a CRC calculation on all fields in the packet except the Preamble, SFD (Start Frame Delimiter), and FCS using a predetermined algorithm. The source host stores the value in the FCS field and transmits it as part of the packet. When the packet is received by the destination host, it performs a CRC test again by using the same algorithm. If the CRC value calculated at the destination host does not match the value in the FCS field, the destination host discards the packet, considering this as a CRC Error.
- Example:
Packets received: 1707183512Packets sent: 2164487593Bytes received: 1031748541320Bytes sent: 1903235134748Receive packets dropped: 1Transmit packets dropped: 0Multicast packets received: 92507460Broadcast packets received: 88183479Multicast packets sent: 161466Broadcast packets sent: 448062Total receive errors: 1Receive length errors: 0Receive over errors: 0Receive CRC errors: 1
- Checksum Error:
- In some use cases, "Total receive errors" counter doesn't provide any detail of the error. It is recommended to review driver's private statistics. Some drivers may report errors in "csumErr" counter, which will be aggregated into "Total receive errors".
csumErr may occur due to one of the following reasons:- Layer3 checksum error, introduced by hardware issue (typically cabling),
- Ingress packet has Layer4 checksum (packet is encapsulated, e.g. in vxlan or overlay),
- Receive packet rate on the vmnic is too high and the hardware is unable to perform the checksum calculations in timely manner. This traffic will be passed on to the ESXi for delivery to the VM/vmknic (which is expected to do it's own checksum validations), but this traffic will be declared as error.
- Example:
Packets received: 460935505Packets sent: 765907671Bytes received: 1185639933809Bytes sent: 1891036423582Receive packets dropped: 0Transmit packets dropped: 0Multicast packets received: 6340187Broadcast packets received: 1558633Multicast packets sent: 844076Broadcast packets sent: 247884Total receive errors: 28910rxq0: totalPkts=87671882 totalBytes=38185291951 nonEopDescs=0 allocRxBufFail=0 csumErr=11882rxq1: totalPkts=4226250 totalBytes=4885323247 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq2: totalPkts=705583 totalBytes=432939698 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq3: totalPkts=1257818 totalBytes=1888832183 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq4: totalPkts=0 totalBytes=0 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq5: totalPkts=0 totalBytes=0 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq6: totalPkts=0 totalBytes=0 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq7: totalPkts=0 totalBytes=0 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq8: totalPkts=38806293 totalBytes=19133491524 nonEopDescs=0 allocRxBufFail=0 csumErr=4778rxq9: totalPkts=56875809 totalBytes=20499650948 nonEopDescs=0 allocRxBufFail=0 csumErr=4697rxq10: totalPkts=89970461 totalBytes=113033495851 nonEopDescs=0 allocRxBufFail=0 csumErr=7554rxq11: totalPkts=27056893 totalBytes=139387254694 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq12: totalPkts=2661580 totalBytes=1359078483 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq13: totalPkts=148570974 totalBytes=789839228623 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq14: totalPkts=684135 totalBytes=320509941 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq15: totalPkts=1636002 totalBytes=1949671803 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq16: totalPkts=7216110 totalBytes=48745812682 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq17: totalPkts=69642 totalBytes=101531218 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq18: totalPkts=50 totalBytes=16614 nonEopDescs=0 allocRxBufFail=0 csumErr=0rxq19: totalPkts=1424929 totalBytes=2134237438 nonEopDescs=0 allocRxBufFail=0 csumErr=0
- In some use cases, "Total receive errors" counter doesn't provide any detail of the error. It is recommended to review driver's private statistics. Some drivers may report errors in "csumErr" counter, which will be aggregated into "Total receive errors".
Resolution
If any NICs show errors, consult with the hardware vendor to troubleshoot the physical NIC errors as they are only reported to the ESXi hosts.
Below are further troubleshooting steps that may help while investigating physical NIC errors:
- Check that the driver/firmware of the vmnic is up to date.
- To check the driver, follow Determining Network/Storage firmware and driver version in ESXi (323110).
- For more information on drivers and firmware, see FAQ: Recommendation for Driver/Firmware (318542)
- It is possible to mitigate FIFO or Missed errors by increasing the Rx buffer ring size on the physical NIC.
- Increasing the Rx and Tx values on the hardware side and within the guest OS can significantly enhance VM performance, especially in high I/O environments. However, the effectiveness of this adjustment depends on factors such as the application, operating system, and hardware in use. Therefore, the recommendations should come from the respective application, OS, and hardware vendors to ensure compatibility and optimal performance.
- Use the following commands to check the maximum (preset) ring buffer size and current ring buffer size.
$ esxcli network nic ring preset get -n vmnic# $ esxcli network nic ring current get -n vmnic#
Information Regarding Physical NIC Ring Buffers:
A ring buffer is used to allocate a section of memory which is like a temporary holding area for packets whose packet rate may be so high that the code required to process them has trouble keeping up.
Each type of network adapter has a preset maximum which is determined by the device driver. This is revealed by the above command featuring preset.
The preset maximum is usually higher than the default setting which is allocated when ESXi is installed. This is revealed by the above command featuring current.
Below is an example of the output from the buffer commands above (the output may differ between vendors):
Current Ring Size: RingInfo: RX: 1023 RX Mini: 0 RX Jumbo: 0 TX: 1023
Preset Maximum Ring Size: RingInfo: Max RX: 4095 Max RX Mini: 0 Max RX Jumbo: 0 Max TX: 4095
With the development of higher and higher NIC speeds and physical switches, and applications which involve higher volumes of TCP/IP packets and packet rates, it is becoming more common to see situations where increasing the physical uplink ring buffer sizes to the preset maximums, will reduce the possibility of drivers being unable to cope with high packet arrival rates, however confirmation from the NIC vendor is recommended prior to making any changes.
Command to increase both rx and tx:
esxcli network nic ring current set -n vmnic4 -r 4096 -t 4096
For receive ring buffer only:
esxcli network nic ring current set -n vmnic4 -r 4096
Max value will depend on the one got from the get command:
esxcli network nic ring preset get -n vmnicX
NOTE : Confirm with hardware vendor the recommended value to be set.
Additional Information
There may be a log message that populates for vSAN environments stating "High pNic error rate detected" which takes data from a specific group of the counters listed above that can impact vSAN performance. Please refer to Alarm about high pNIC error rate being detected for more information if this error message is present.
Feedback
