High pNIC error detected on two different vmnics

Products

• VMware vSphere ESXi

Issue/Introduction

• Intel X710 network adapters reporting Receive Length Errors.
• Error message thrown "High pNic RX Generic error rate detected on vmnic..."
• The NIC public statistics show Receive Length Errors.
  esxcli network nic stats get -n vmnicX
  
  
  

Environment

VMware Vsphere 

Cause

• If an Ethernet packet is received with a packet length larger than 64 bytes and the Length/Type field contains a value that is smaller than the number of bytes remaining in the packet, an Rx length error is detected.
• This type of packet could be generated if a short packet is padded to 64 bytes and then a VLAN tag is inserted without removing any padding bytes.
• When an Rx length error is detected, the packet is considered a bad packet and is discarded.
• Packets with the structure defined above are dropped.
• These packets are often local diagnostic packets that might be discarded by the software anyway.
• The driver reports the error count to the OS, possibly resulting in a high error rate that is reported by the OS.
  
  

Resolution

• Validate that the driver and firmware are up to date and on the VMware compatibility list.
• The driver should not report the Rx length error count to the OS. This is implemented in Intel drivers starting from Release 28.2.
• Contact the NIC vendor for further investigation or replacement of the hardware.

Additional Information

• Receive over errors
  • The packets that are discarded by the hardware buffer of the card.
    • The total receive errors are any CRC errors as well as possibly values from the other counters. 
  • Example:
       Packets received: 1707183512
       Packets sent: 2164487593
       Bytes received: 1031748541320
       Bytes sent: 1903235134748
       Receive packets dropped: 1
       Transmit packets dropped: 0
       Multicast packets received: 92507460
       Broadcast packets received: 88183479
       Multicast packets sent: 161466
       Broadcast packets sent: 448062
       Total receive errors: 9464
       Receive length errors: 0
       Receive over errors: 9463
       Receive CRC errors: 1
       Receive frame errors: 0
       Receive FIFO errors: 0
       Receive missed errors: 0
       Total transmit errors: 0
       Transmit aborted errors: 0
       Transmit carrier errors: 0
       Transmit FIFO errors: 0
       Transmit heartbeat errors: 0
       Transmit window errors: 0