Error message observed on Zabbix monitoring tool - Problem: Interface vmxnet3 Ethernet Adapter(Ethernet0): High error rate (>2 for 5m)

Products

• VMware vSphere ESXi

Issue/Introduction

The following error is observed on the monitoring tool Zabbix for VMs: Problem: Interface vmxnet3 Ethernet Adapter(Ethernet0): High error rate (>2 for 5m)

Environment

VMware vSphere 

Cause

The reported errors are due to buffer exhaustion and saturated ring buffers on the affected VMs

The above can be validated using: 

1. For a given virtual machine, determine the ESXi host on which it is running, and log into the ESXi host via SSH or a KVM with root privileges.
2. Assuming an example of "VMNAME", run the command:  net-stats -l | grep -i VMNAME
3. The output might look something like the following:

<PortNumber>       5    9    <Switch Name>    <MAC Address>    VMNAME.eth0

4. Run a command such as the following to check the 1st ring buffer stats:

vsish -e get /net/portsets/<Switch Name>/ports/<PortNumber>/vmxnet3/rxSummary | grep "1st ring"

5. The output might look something like the following: 

 1st ring size:512
   # of times the 1st ring is full:276

6.Another command that will show insight would be:

vsish -e get /net/portsets/<Switch Name>/ports/<PortNumber>/vmxnet3/rxSummary | grep "running out of buffers"

7. The output might look something like the following:
     running out of buffers:3198

Resolution

• Increase the ring buffer size on the VMXnet3 adapter
• You can refer the following KB: Large packet loss (dropped packets in VM) in the guest OS using VMXNET3 in ESXi

Additional Information

net-VM-vmxnet3