ESXi 7.0.x reporting out of memory errors for vmware-usbarbitrator, sfcb-vmw-ipmi, sensord, or vdpi in vmkernel.log / syslog on HPE servers

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The issue is primarily seen on HPE Servers
• In some cases, the affected ESXi host appears as not responding in vCenter Inventory
• VMs could restart after being failed over by vSphere HA
• The ESXi host may appear normal, you can connect via SSH, run esxcli commands etc
• vmkernel logs are contain a high number of "Cannot allocate memory" messages like the below:

cpu55:2099016)WARNING: UserObj: 4914: vmware-usbarbitrator: Poll on fd 10 failed: Out of memory
error hostd[2100397] [Originator@6876 sub=Libs] BmcInfo: poll(43), failed 12 Cannot allocate memory
error hostd[2100572] [Originator@6876 sub=Default] Node 0: poll(fd=32), errno=12 Cannot allocate memory

cpu44:2116046)WARNING: UserObj: 4914: vdpi: Poll on fd 15 failed: Out of memory
cpu22:2116045)WARNING: UserObj: 4914: vdpi: Poll on fd 12 failed: Out of memory

 

 

• Syslog may contain memory errors for sfcb-vmw_ipmi and/or sensord

sfcb-vmw_ipmi[2115198]: Node 0: poll(fd=75), errno=12 Cannot allocate memory
sfcb-vmw_ipmi[2115198]: Node 0: poll(fd=75), errno=12 Cannot allocate memory
sfcb-vmw_ipmi[2115198]: Node 0: poll(fd=75), errno=12 Cannot allocate memory
sfcb-vmw_ipmi[2115198]: Node 0: poll(fd=75), errno=12 Cannot allocate memory

sensord[2099995]: recv_reply: poll(0), failed 12 Cannot allocate memory
sensord[2099995]: recv_reply: poll(0), failed 12 Cannot allocate memory

 

• You may have network latency for the vm's if vdpi is out of memory.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 7.0.3

 

Cause

The memory allocation issue is being caused by the HPE SUT (Integrated Smart Update Tool) utility.   This is not a VMware issue but can affect many different processes in ESXi. 

Resolution

VMware and HPE Engineering teams are working together to find the exact root cause and provide a permanent solution in a future release of the HPE SUT utility.

For more details on this issue and prevent it from impacting your environment, please engage the HPE support.

Workaround:

As a workaround to prevent this memory allocation issue, you can uninstall the HPE SUT component.

• De-Register the SUT utility:

  # sut -deregister

• Uninstall the SUT component:

  # esxcli software component remove –n sutComponent

Additional Information

To verify if the SUT component is installed on the ESXi, use the following command:

# esxcli software component list | grep -i sut

The output should will show:

sutComponent Integrated Smart Update Tool