sfcb process crashes with "signal:6 exitCode:0 coredump:enabled" error on ESXi 6.5 hosts

Products

• VMware vSphere ESXi

Issue/Introduction

To resolve the HPE provider crash.

Symptoms:

• The sfcb process crashes in ESXi 6.5 host and creates zdump files in /var/core/
• In /var/run/log/hostd.log, you see messages similar to:

2016-12-29T00:22:55.455Z info hostd[CB81B70] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
--> key = 218190868,
--> chainId = 0,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> name = "esxi.hostname.local",
--> host = 'vim.HostSystem:ha-host'
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.application.core.dumped",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "1",
--> value = "/bin/sfcbd"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "2",
--> value = "309"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "3",
--> value = "/var/core/sfcb-smx-zdump.002"
--> }
--> ],
--> objectId = "ha-host",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null
--> }
2016-12-29T00:22:55.455Z verbose hostd[CB81B70] [Originator@6876 sub=PropertyProvider] RecordOp REMOVE: latestPage[213], session[52f2e39a-eaf9-e8c6-1154-e62673660715]52345a88-0c77-9f7a-c104-51afbb40364f. Applied change to temp map.
2016-12-29T00:22:55.455Z verbose hostd[CB81B70] [Originator@6876 sub=PropertyProvider] RecordOp ADD: latestPage[223], session[52f2e39a-eaf9-e8c6-1154-e62673660715]52345a88-0c77-9f7a-c104-51afbb40364f. Applied change to temp map.
2016-12-29T00:22:55.455Z verbose hostd[CB81B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2016-12-29T00:22:55.456Z info hostd[CB81B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 223 : An application (/bin/sfcbd) running on ESXi host has crashed (309 time(s) so far). A core file might have been created at /var/core/sfcb-smx-zdump.002.

• In /var/run/log/vmkernel.log, you see messages similar to:

2016-12-29T00:22:53.490Z cpu2:316641)User: 3089: sfcb-smx: wantCoreDump:sfcb-smx signal:6 exitCode:0 coredump:enabled
2016-12-29T00:22:53.595Z cpu0:316641)UserDump: 3024: sfcb-smx: Dumping cartel 316636 (from world 316641) to file /var/core/sfcb-smx-zdump.002 ...
2016-12-29T00:22:55.454Z cpu2:316641)UserDump: 3172: sfcb-smx: Userworld(sfcb-smx) coredump complete.

Note: These log excerpts are an example. Date, time, and environmental variables may vary depending on your environment. Additional sfcb providers other than hpe-smx-provider may also exhibit the same behavior. 

Environment

VMware vSphere ESXi 6.5

Cause

This issue occurs when the sfcb process runs out of available memory in the resource pool.

Resolution

This issue is resolved in HPE VMware ESXi 6.5 (July 2017) custom ISO image. For more information refer to HPE customer advisory 

Workaround:
To workaround of not getting indications after a provider crashes, restart the HPE Insight Management WBEM Providers, by performing any of the following procedures:

• Run the command "/etc/init.d/sfcbd-watchdog restart".

• Use the console to restart the management agents.

• Use vCenter to restart the management agents

Additional Information

ESXi 6.5 主机上出现“sfcb-smx: wantCoreDump:sfcb-smx signal:6 exitCode:0 coredump:enabled”错误
ESXi 6.5 ホストで「sfcb-smx: wantCoreDump:sfcb-smx signal:6 exitCode:0 coredump:enabled」エラーが発生する

 

Impact/Risks:
Restarting ESXi management agents will momentarily shows ESXi host as disconnected.

Warining: If LACP is configured on the vSAN network, do not restart management agents on ESXi hosts running vSAN.

• Restarting the management agents may impact any tasks that are running on the ESXi host at the time of the restart.
• Check for any storage issues before restarting the Host deamon hostd service or services.sh