Troubleshooting network receive traffic faults and other NIC errors in ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

You may notice that there are many receive FIFO errors, or other errors, on the ESXi host physical NICs:

esxcli network nic stats get -n vmnic0

NIC statistics for vmnic0
...
Receive FIFO errors: 505563312

Receive missed errors: 34871289
..

Note: The above output is a snippet from an example for vmnic0. Specify the vmnic# corresponding to the one you are troubleshooting. Normally, the above values would be either zero or very close to zero. If the numbers are small in comparison to the total number of packets for that NIC, then you would only want to consider this resolution in situations where the number increases significantly during the incident you are troubleshooting.

Note: This solution is for ESXi host physical NICs. For a similar solution regarding the guest virtual NICs, see Large packet loss in the guest OS using VMXNET3 in ESXi (324556). Please note that these are physical NICs and the hardware vendor should troubleshoot the physical NICs when errors are present.

 

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Resolution

If any of the NICs show errors with the above command, open a case with the hardware vendor to troubleshoot the physical NIC errors and follow these steps:

1. Check that the driver/firmware of the vmnic is up to date. To check the driver, follow Determining Network/Storage firmware and driver version in ESXi (323110).  For more information on drivers and firmware, see FAQ: Recommendation for Driver/Firmware (318542)
2. If the driver is up to date, you may be able to avoid FIFO or Missed errors by increasing the Rx buffer ring size on the physical NIC.

Note: These changes impact network adapter performance and must be validated by the hardware vendor who supplies the network adapter.

FIFO or Missed errors (one or the other, not necessarily both) will increment and accumulate if physical NIC is not able to handle the peak load of incoming packets with current rx ring buffer size.

Use the following commands to check the current rx ring buffer size and maximum (preset):

esxcli network nic ring preset get -n vmnicX
esxcli network nic ring current get -n vmnicX

Review the output and check the current rx ring buffer size compared to the preset maximum rx ring buffer size the NIC supports.

As an example, for a NIC that supports a preset maximum of 4080, you could first try increasing this value to 1020, and then 2040 (if 1020 was not enough to prevent errors), and finally 4080.

Note: Do the following steps with the ESXi host in maintenance mode, to avoid any potential production impact. Use an out-of-band connection (iLO, DRAC, etc.), when changing the rx ring buffer size to ensure you can change it back if the connection is disrupted because of the change.
(Use "-t number" to change the transmit buffer if needed)

esxcli network nic ring current set -n vmnicX -r number

Note: These changes impact network adapter performance and must be validated by the hardware vendor who supplies the network adapter. If there are any questions on the above commands, refer to the hardware vendor.