High pNic error rate detected due to high "Receive length errors" on host uplinks.

Products

• VMware vSphere ESXi
• VMware vSAN

Issue/Introduction

"High pNic error rate detected. Check the host's vSAN performance view for details" alarms are reported in vCenter. The affected VMNICs' stats, which can be listed by running "esxcli network nic stats get -n vmnic<>" command, show a high count of "Receive length errors" resulting in a high "Total Rx errors" count for the same VMNIC which in turn trigger the alarms.

Environment

VMware vSphere ESXi

Cause

The "Receive length errors" counter can increment upon the receipt of malformed packets. This specifically occurs with packets where the length reported by the lower-level Protocol Data Unit (PDU) is smaller than the length reported by the higher-level PDU. Example:

Notice that the frame length (821) is less than the length specified in the UDP header (17067). This is marked as malformed by Wireshark as the frame length is expected to be more than the higher layer protocols like UDP, TCP, IP, etc. that are encapsulated in the frame.

Note: The Frame.length and the Length (UDP) columns in the screenshot above were added and are not available in the default Wireshark view. This can be added by right clicking on the length field for any header and selecting "Apply as Column".  

Resolution

A packet capture can be performed on the affected uplink(s) to identify malformed packet(s). To perform an unfiltered capture for inbound packets use the following command:

pktcap-uw --uplink <vmnicX> --dir 0 --ng -o </datastore/directory/file.pcapng>

Try to limit the capture size by running it for as little time as possible. To ensure you've captures at least a few malformed packets before stopping it, you can watch the NIC stats in a separate terminal window to the same hypervisor:

watch esxcli network nic stats get -n vmnic<>

Look for "Receive length errors", once this counter has incremented a few times, you can stop the packet capture.

Once the original issue is addressed so that we are no longer receiving the malformed packets, the VMNIC stats counters can be reset to clear the alarm by rebooting the host. KB vSAN -- Alarm about high pNIC error rate being detected can be referenced for more information.