ESXi 7.0.x reporting out of memory errors for vmware-usbarbitrator, sfcb-vmw-ipmi, sensord, or vdpi in vmkernel.log / syslog on HPE servers

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You may have network latency for the vm's if vdpi is out of memory.
• The issue is primarily seen on HPE Servers
• In some cases, the affected ESXi host appears as not responding in vCenter Inventory
• VMs could restart after being failed over by vSphere HA
• The ESXi host may appear normal, you can connect via SSH, run esxcli commands etc
• vmkernel logs are contain a high number of "Cannot allocate memory" messages like the below:

cpu55:2099016)WARNING: UserObj: 4914: vmware-usbarbitrator: Poll on fd 10 failed: Out of memory

error hostd[2100397] [Originator@6876 sub=Libs] BmcInfo: poll(43), failed 12 Cannot allocate memory

error hostd[2100572] [Originator@6876 sub=Default] Node 0: poll(fd=32), errno=12 Cannot allocate memorycpu44:2116046)WARNING: UserObj: 4914: vdpi: Poll on fd 15 failed: Out of memory

cpu22:2116045)WARNING: UserObj: 4914: vdpi: Poll on fd 12 failed: Out of memory

 

• Syslog may contain memory errors for sfcb-vmw_ipmi and/or sensord

sfcb-vmw_ipmi[2115198]: Node 0: poll(fd=75), errno=12 Cannot allocate memory
sensord[2099995]: recv_reply: poll(0), failed 12 Cannot allocate memory

Environment

VMware vSphere ESXi (All Versions)

 

Cause

The memory allocation issue is being caused by the HPE iSUT (Integrated Smart Update Tool) utility.

This is not a VMware issue but can affect many different processes in ESXi.

Resolution

For more details on this issue and prevent it from impacting your environment, please engage HPe support.

This has been fixed in ESXi versions 7.0U3q and 8.0U2b and higher as well as in HPe iSUT version 2.9.0

If upgrade is not possible at this time follow the below workaround

 

Workaround:

As a workaround to prevent this memory allocation issue, you can uninstall the HPe iSUT component by following these steps:

1.) It is preferred to put the Host in Maintenance Mode

2.) Verify that the HPE iSUT component is installed by running without "#": 

# esxcli software component list | grep -i sut

The output should show: sutComponent Integrated Smart Update Tool

Example: 

 

3.) De-Register the iSUT utility by running without "#": 

# sut -deregister

4.) Uninstall the iSUT component by running without "#": 

# esxcli software component remove –n sutComponent

5.) You should see - Example:

 

6.) Proceed with rebooting of the Host

7.) When the Host is back from reboot but still in Maintenance Mode verify that the HPE iSUT component is gone by running without "#": 

# esxcli software component list | grep -i sut

8.) If all is looking good, put the Host out of Maintenance Mode

Additional Information

Advisory: (Revision) Integrated Smart Update Tools (iSUT) - Memory Leak May Occur When Running VMware ESXi 6.x/7.x and iSUT Versions 2.4.0 Up To 2.8.0