"Network: Out of memory" Alerts from ESXi host in VCF Operations for Logs

Products

• VMware vSphere ESXi

Issue/Introduction

VCF Operations for Logs triggers repeated alerts for "Network: Out of memory" on a specific ESXi host. When looking at the ESXi host in VCenter and iDrac there are no alerts for this. 

While vCenter Server and hardware management consoles (e.g., iDRAC) may not display active alarms, the ESXi vmkwarning logs indicate heap memory exhaustion. The alert text often references generic NetQueue or Jumbo Frame issues, but the underlying logs point to specific module failures.

VCF Operations for Logs has been configured to provide an alert when certain strings are seen in logs. In this example, the string configured is "Network: Out of memory".

Symptoms include:

• Log Insight alerts stating: Log Insight found the following 1 group matching the criteria for alert "Network: Out of memory"

• Warnings in /var/run/log/vmkwarning.log indicating NetPktHeapLow exhaustion.

• References to dellism (Dell iDRAC Service Module) or vusb0 failing to acquire resources.

Validate Log Pattern:

Check /var/run/log/vmkwarning.log on the affected ESXi host. Confirm the presence of dellism or vusb0 errors coinciding with NetPktHeapLow exhaustion:

WARNING: vmkusb: udev ... invalid state 5: Failure
WARNING: LinuxThread: 411: dellism: Error cloning thread: -28 (bad0081)
WARNING: Heap: 3892: Could not allocate 2048 bytes for dynamic heap NetPktHeapLow. Request returned Out of memory

 

NOTE:  VCF Operations for Logs was formerly named Aria Operations for Logs, and prior to that was named vRealize Log Insight (vRLI).

Environment

VMware ESXi

Dell PowerEdge Hardware

Dell iDRAC Service Module (iSM) installed

Cause

The issue is caused by the third-party Dell iDRAC Service Module (dellism) or the USB arbitrator service (vusb0) failing to manage memory correctly, leading to exhaustion of the NetPktHeapLow dynamic heap. This prevents the ESXi host from allocating necessary network packet buffers.

The Log Insight alert is a generic response to the "Out of memory" string, but the root cause is specific to the vendor module's thread cloning and memory allocation failures.

Resolution

The errors originate from a third-party hardware management module.

• Engage Vendor Support:

  If the issue persists on the latest firmware, engage Dell Technical Support to investigate the Dell iDRAC Service Module (dellism) process behavior and potential hardware-level USB arbitration issues.