VMXNET3 "out of buffers" packet drops on Windows VMs in ESXi 8.0

Products

• VMware vSphere ESXi

Issue/Introduction

• Packet drops at the VMXNET3 interface.
• High memory usage in the guest OS associated with event logging services.
• Degraded application performance (e.g., SQL audit log processing).
• Exact Error/Counter: ESXi vsish statistics report a "running out of buffers" counter incrementing (e.g., running out of buffers: 274190).

Environment

VMware ESXi 8.0

Cause

The issue is triggered by CPU contention on the ESXi host. While the VMXNET3 ring buffers act as a temporary queue for incoming packets, they are exhausted when the VM's assigned CPU resources are insufficient to process the data at the rate it is being delivered. When the ring buffers are full, the host is forced to drop subsequent packets, leading to the "running out of buffers" error.

Resolution

To resolve this issue, the processing capacity for the VMXNET3 receive path must be stabilized through resource allocation and buffer adjustments.

1. Enable CPU Reservation: Assign a CPU reservation to the affected VM to ensure it has guaranteed CPU cycles to process the network queue, even during periods of host-wide contention.
   • Right-click the VM in the vSphere Client and select Edit Settings.
   • Expand CPU and set the Reservation to a value that meets the workload requirement.
   • Increase Guest OS Ring Buffer Size: Increasing the Rx ring buffer size within the Windows guest provides a larger cushion for packet bursts.
2. Open Device Manager in the Windows guest.
   • Locate the VMXNET3 Ethernet Adapter, right-click, and select Properties.
   • Under the Advanced tab, find Small Rx Buffers, Rx Ring #1, and Rx Ring #2 Size.
   • Increase these values to their maximum allowed limits (this will increase RAM usage on the VM)

If CPU reservations cannot be applied due to cluster capacity constraints, increasing the ring buffer sizes in the guest OS (Step 2 above) may provide modest, temporary improvement, though it may not entirely eliminate drops during peak contention.