Receive packet errors in a vSAN cluster that move from server to server but always on vmnic0

Products

• VMware vSAN

Issue/Introduction

Inconsistent cluster host level alerts for "High pNic error rate detected. Check the host's vSAN performance view for details" & "RX missed error rate reaches 1‰ on Physical Adapter vmnic0"

These symptoms can represent a multitude of NIC errors visible in the following esxcli output.  Examples below show receive miss errors on vmnic0 and CRC errors for vmnic4 from the same host, where each physical vmnic is an uplink for a different virtual switch and the same symptoms are present across all hosts in a vSAN cluster:

# esxcli network nic stats get -n vmnic0
   NIC statistics for vmnic0:
      Packets received: 9018105094
      Packets sent: 4881349417
      Bytes received: 7117432651253
      Bytes sent: 41722914454079
      Receive packets dropped: 0
      Transmit packets dropped: 0
      Multicast packets received: 88624536
      Broadcast packets received: 103076535
      Multicast packets sent: 1212186
      Broadcast packets sent: 1208592
      Total receive errors: 0
      Receive length errors: 0
      Receive over errors: 0
      Receive CRC errors: 0
      Receive frame errors: 0
      Receive FIFO errors: 0
      Receive missed errors: 22750
      Total transmit errors: 0
      Transmit aborted errors: 0
      Transmit carrier errors: 0
      Transmit FIFO errors: 0
      Transmit heartbeat errors: 0
      Transmit window errors: 0

# esxcli network nic stats get -n vmnic4
   NIC statistics for vmnic4:
      Packets received: 1082176345
      Packets sent: 560623522
      Bytes received: 7875423632576
      Bytes sent: 1378107493201
      Receive packets dropped: 0
      Transmit packets dropped: 0
      Multicast packets received: 28285908
      Broadcast packets received: 30122854
      Multicast packets sent: 181898
      Broadcast packets sent: 341667
      Total receive errors: 7058
      Receive length errors: 193
      Receive over errors: 0
      Receive CRC errors: 6865
      Receive frame errors: 0
      Receive FIFO errors: 0
      Receive missed errors: 0
      Total transmit errors: 0
      Transmit aborted errors: 0
      Transmit carrier errors: 0
      Transmit FIFO errors: 0
      Transmit heartbeat errors: 0
      Transmit window errors: 0

Environment

8.0 U3

Cause

In this specific scenario, the cause is a combination of CRC errors on one uplink and receive miss errors on another uplink.  As described by the following KB, CRC errors indicate physical cabling issues, while receive miss errors indicate a vmnic ring buffer set too low.
https://knowledge.broadcom.com/external/article?articleNumber=341594

Which alerts are thrown are dependent on the individual error rate warning/critical thresholds which are described in the following KB.
https://knowledge.broadcom.com/external/article?articleNumber=312096

Resolution

Investigate CRC errors at the physical/cabling layer and increase ring buffer sizes to eliminate receive miss errors