smx-provider crashes due to memory allocation issues in on HPE hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• In vSphere Client, the following event is seen:

“An application (/bin/sfcbd) running on ESXi host has crashed (1 time(s) so far). A core file might have been created at /var/core/sfcb-smx-zdump.000.”. 

• When reviewing the vmkernel.log, located in /var/run/log/, the following entries are seen:
  
  [YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 478: UserWorld 'sfcb-smx' with cmdline '/bin/sfcbd'
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 489: uw.2251206 (655346) extraMin/extraFromParent: 329/329, smx (12096) childEmin/eMinLimit: 17739/17920
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 471: Admission failure in path: smx/sfcb-ProviderMa.2251206/uw.2251206
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 478: UserWorld 'sfcb-smx' with cmdline '/bin/sfcbd'
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)MemSchedAdmit: 489: uw.2251206 (655346) extraMin/extraFromParent: 329/329, smx (12096) childEmin/eMinLimit: 17739/17920
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)User: 3173: sfcb-smx: wantCoreDump:sfcb-smx signal:6 exitCode:0 coredump:enabled
[YYYY-MM-DDTHH:MM:SS] cpu31:2251208)UserDump: 3110: sfcb-smx: Dumping cartel 2251206 (from world 2251208) to file /var/core/sfcb-smx-zdump.000 ...


• In the syslog.log file, the following errors are seen:
  
  sfcb-vmware_base[2110110]: tool_mm_realloc_or_die: memory re-allocation failed(orig=364000 new=364800 msg=Cannot allocate memory, aborting
sfcb-vmw_ipmi[2291550]: tool_mm_realloc_or_die: memory re-allocation failed(orig=909200 new=909600 msg=Cannot allocate memory, aborting
  
  
• The host is getting constantly disconnected from the vCenter due to the request made by the localhost 127.0.0.1.
  
  
• Failed to read buffer from stream; <io_obj p:0x000000a79aa27c60, h:35, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 30847'>> e: 104(Connection reset by peer), async: true, duration: 0msec
  
  
• Failed to shutdown socket; <io_obj p:0x000000a79aa27c60, h:35, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 30847'>>, e: 104(shutdown: Connection reset by peer [system:104])

Environment

7, 8

Cause

This is caused when the HPE's smx-provider runs out of memory. The smx-provider is the WBEM provider for the HPE SmartArray controller. Like most daemons on an ESXi host, this provider runs in its own resource pool and can allocate only as much memory as the pool configuration allows.

A service that does not have its own specific resource pool configuration gets a default memory limit of 70 Mbyte. Depending on the service, this can be insufficient.

Attempts to allocate memory above the configured limit will cause the service to crash and trigger the events mentioned above.

Resolution

Please note that Hewlett Packard Enterprise have decided to no longer maintain the smx-provider VIB. The official recommendation to solve this issue is therefore to uninstall the VIB.
Be aware that uninstalling the SMX provider VIB requires a reboot of the ESXi host, therefore a maintenance window should be planned.

To uninstall the SMX provider

1. Connect to the ESXi host per SSH
2. Depending on the ESXi version, run the following command to uninstall the VIB:
   1. for ESXi 6.5 or 6.7:
      
      # esxcli software vib remove -n smx-provider
      
      
   2. For ESXi 7.x and later:
      
      # esxcli software component remove -n smxProvider

NOTE: in some cases, the esxcli software component remove command can fail on ESXi 7.x. If that happens, please use the esxcli software vib remove command on ESXi 7.x similar to ESXi 6.x

3. To finalize the removal, reboot the ESXi host

NOTE: In VMware vSphere ESXi7.0U3i version it sets the default size of vmware_base group to 120Mb. For more details refer to VMware vSphere ESXi7.0U3i release notes.

Additional Information

See Advisory: (Revision) HPE Smart Array SR Gen10 Controllers - Additional Disks Are Displayed In ESXi /vCenter GUI and smx-provider Errors Are Logged On Platforms Configured With Controllers Using FW v3.53 (Or Later)  for more details

Impact/Risks:
The smx-provider provides health information from the HPE SmartArray controller. Removing the VIB might cause certain health information for the controller to become unavailable in the vSphere Client hardware health tab for the ESXi hosts.