smx-provider crashes due to memory allocation issues in ESXi 6.x/7.x on HPE hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• In vSphere Client the following event is seen:

“An application (/bin/sfcbd) running on ESXi host has crashed (1 time(s) so far). A core file might have been created at /var/core/sfcb-smx-zdump.000.”. 

• When reviewing the vmkernel.log, located in /var/run/log/, the following entries are seen:
  
  [YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 478: UserWorld 'sfcb-smx' with cmdline '/bin/sfcbd'
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 489: uw.2251206 (655346) extraMin/extraFromParent: 329/329, smx (12096) childEmin/eMinLimit: 17739/17920
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 471: Admission failure in path: smx/sfcb-ProviderMa.2251206/uw.2251206
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 478: UserWorld 'sfcb-smx' with cmdline '/bin/sfcbd'
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 489: uw.2251206 (655346) extraMin/extraFromParent: 329/329, smx (12096) childEmin/eMinLimit: 17739/17920
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)User: 3173: sfcb-smx: wantCoreDump:sfcb-smx signal:6 exitCode:0 coredump:enabled
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)UserDump: 3110: sfcb-smx: Dumping cartel 2251206 (from world 2251208) to file /var/core/sfcb-smx-zdump.000 ...


• In the syslog.log file, the following errors are seen:
  
  sfcb-vmware_base[2110110]: tool_mm_realloc_or_die: memory re-allocation failed(orig=364000 new=364800 msg=Cannot allocate memory, aborting
sfcb-vmw_ipmi[2291550]: tool_mm_realloc_or_die: memory re-allocation failed(orig=909200 new=909600 msg=Cannot allocate memory, aborting

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 6.7

VMware vSphere ESXi 8.0.x

Cause

This is being caused when the HPE's smx-provider runs out of memory. The smx-provider is the WBEM provider for the HPE SmartArray controller. Similar to most of the daemons on an ESXi host, this provider runs in its own specific resource pool and can only allocate as much memory as the pool configuration allows it to.

A service which does not have its own specific resource pool configuration, does get a default memory limit of 70 Mbyte. Depending on the service, this can be insufficient.

Attempts to allocate memory above the configured limit will lead to the service crashing, leading to the events mentioned above.

Resolution

Please note that Hewlett Packard Enterprise have decided to no longer maintain the smx-provider VIB. The official recommendation to solve this issue is therefore to uninstall the VIB.
Be aware that uninstalling the SMX provider VIB requires a reboot of the ESXi host, therefore a maintenance window should be planned.

To uninstall the SMX provider

1. Connect to the ESXi host per SSH
2. Depending on the ESXi version, run the following command to uninstall the VIB:
   1. for ESXi 6.5 or 6.7:
      
      # esxcli software vib remove -n smx-provider
      
      
   2. For ESXi 7.x and later:
      
      # esxcli software component remove -n smxProvider

NOTE: in some cases, the esxcli software component remove command can fail on ESXi 7.x. If that happens, please use the esxcli software vib remove command on ESXi 7.x similar to ESXi 6.x

3. To finalize the removal, reboot the ESXi host

NOTE: In VMware vSphere ESXi7.0U3i version it sets the default size of vmware_base group to 120Mb. For more details refer to VMware vSphere ESXi7.0U3i release notes.

Additional Information

See https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00117054en_us for more details

Impact/Risks:
The smx-provider serves as a data provider for health information from the HPE SmartArray controller. Removing the VIB might result in certain health information for the controller not being visible in the vSphere Client hardware health tab for the ESXi hosts.