ESXi hosts reporting "Out of memory" errors for multiple services on HPe servers

Products

• VMware vSphere ESXi
• VMware vSAN

Issue/Introduction

• All ESXi services such as vmware-usbarbitrator, sfcb-vmw-ipmi, sensord, or vdpi are reporting "Out of memory" in vmkernel.log
• All vSAN services such as clomd, vsantraced, vsantracedUrgen, vsantracedDiag are reporting "Out of memory" in vmkernel.log
• vSAN cluster becomes unmanageable and VMs fail to deploy/create 
• The VMs may face network latency if vdpi is out of memory.
• The issue is primarily seen on HPe Servers.
• In some cases, the affected ESXi host appears as Not Responding in vCenter Inventory.
• VMs can be restarted after being failed over by vSphere HA.
• The ESXi host may appear normal; SSH connection, executing esxcli commands work as expected.
• /var/run/log/vmkernel.log may contain a high number of "Cannot allocate memory" messages like the following:

cpu55:2099016)WARNING: UserObj: 4914: vmware-usbarbitrator: Poll on fd 10 failed: Out of memory

error hostd[2100397] [Originator@6876 sub=Libs] BmcInfo: poll(43), failed 12 Cannot allocate memory

error hostd[2100572] [Originator@6876 sub=Default] Node 0: poll(fd=32), errno=12 Cannot allocate memorycpu44:2116046)WARNING: UserObj: 4914: vdpi: Poll on fd 15 failed: Out of memory

cpu22:2116045)WARNING: UserObj: 4914: vdpi: Poll on fd 12 failed: Out of memory

cpu68:2111246)WARNING: UserObj: 4914: vdpi: Poll on fd 7 failed: Out of memory
cpu55:2112918)WARNING: UserObj: 4914: clomd: Poll on fd 10 failed: Out of memory
cpu63:2099003)WARNING: UserObj: 4914: vsantraced: Poll on fd 9 failed: Out of memory
cpu63:2099003)WARNING: UserObj: 4914: vRNING: UserObj: 4914: clomd: Poll on fd 10 failed: Out of memory
cpu19:2099025)WARNING: UserObj: 4914: vsantracedUrgen: Poll on fd 9 failed: Out of memory
cpu42:2099041)WARNING: UserObj: 4914: vsantracedDiag: Poll on fd 9 failed: Out of memory


• /var/run/log/syslog.log may contain memory errors for sfcb-vmw_ipmi and/or sensord

sfcb-vmw_ipmi[2115198]: Node 0: poll(fd=75), errno=12 Cannot allocate memory
sensord[2099995]: recv_reply: poll(0), failed 12 Cannot allocate memory

Environment

• VMware vSphere (ALL Versions)
• VMware vSAN (All Versions)
• HPe servers
• HPe iSUT (Integrated Smart Update Tool) utility installed

Cause

The memory allocation issue is being caused by the HPE iSUT (Integrated Smart Update Tool) utility.

This is not a VMware issue, but may impact different processes/services in ESXi/vSAN.

Resolution

For more details on this issue and to prevent it from impacting the environment, please engage HPe support.

This issue has been resolved in HPE iSUT version 2.9.0 (or later). 

 

Workaround:

As a workaround to prevent this memory allocation issue, you can uninstall the HPE iSUT component by following these steps:

• It is preferred to put the host in Maintenance Mode.
• Verify that the HPE iSUT component is installed with the command:
  esxcli software component list | grep -i sut

The output should show: sutComponent Integrated Smart Update Tool

Example:

• De-register the iSUT utility by running the command:
  sut -deregister
• Uninstall the iSUT component by running the command:
  esxcli software component remove –n sutComponent

Expected Output:

• Proceed with the reboot of the ESXi host.
• Once the ESXi host is up after the reboot, verify that the HPE iSUT component is removed by running the command:
  esxcli software component list | grep -i sut
• Once validated that the vib is no longer installed in the host, put the host out of Maintenance Mode.

Additional Information

Advisory: (Revision) Integrated Smart Update Tools (iSUT) - Memory Leak May Occur When Running VMware ESXi 6.x/7.x and iSUT Versions 2.4.0 Up To 2.8.0