ESXI Host is not responding

Products

• VMware vCenter Server

Issue/Introduction

ESXI host is not responding, rebooting the ESXI host recovers the host to a responding state.

There is a sfcb-dcism-zdump in the /var/core directory of the ESXI host.

The Dellism VIB is installed on the ESXI host, this can be confirmed by running the following command on the ESXI Host: esxcli software vib list

 

Environment

ESXI 8.0

vCenter 8.0

Cause

According to the /etc/vmware/daemon/config/dellism.json on the ESXI host, the Dellism VIB is responsible for the following:

"DESCRIPTION": "iDRAC Service Module (iSM) is a lightweight piece of software that runs on the Server and complements iDRAC with monitoring information from the OS."

The ESXI Host is not responding because hostd is failing to respond as the dellism module, which is the ESXi module to communicate with the iDRAC, is seen to be requesting more memory than being assigned to the module for example:

vmkernel.log logs the following:

2024-07-23T03:35:06.564Z In(182) vmkernel: cpuX:2100156)started from 'sh' 2100144 with cmdline '/bin/sh /usr/lib/vmware/daemonMgmt/bin/watchdog.sh -t 100 -n -s dellism-dellism /opt/dellism/bin/dellism', parent 2097xxx
2024-07-23T03:35:06.564Z In(182) vmkernel: cpu1:2100156)started from 'init' 2097725 with cmdline '/bin/init', parent 0
2024-07-23T03:35:06.564Z In(182) vmkernel: cpuX:2100156)uw.2100151 (12513) requires 132 KB, asked 132 KB from dellism (12062) which has 102xxx KB occupied and 112 KB available.
2024-07-23T03:35:06.564Z In(182) vmkernel: cpuX:2100xxx)Admission failure in path: host/vim/vmvisor/dellism:dellism.2100xxx:uw.2100xxx

vmkwarning.log shows the other ESXI core modules are starting to report out of memory:

2024-07-23T03:33:29.773Z Wa(180) vmkwarning: cpuXX:2100xxx)WARNING: World: 3229: Could not allocate new world handle for world ID: 3143xxx: Out of memory 2024-07-23T03:33:29.773Z Wa(180) vmkwarning: cpuXX:2100xxx)WARNING: LinuxThread: 411: dellism: Error cloning thread: -12 (bad0014)



The above symptoms leads to the Host becoming unresponsive.

Resolution

Engage the hardware vendor (Dell) to investigate further for example checking the version of the Dellsim VIB or if there is any known issues with the VIB.