Esxi hosts becoming unresponsive due to vendor images with customized drivers where ImageConfigManager consumes more RAM than allocated

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host shows as "not responding" in vCenter Server
• Custom images provided by our vendors are used to deploy the hosts
• In the /var/log/vmkernel.log file of the ESXi host, below events are seen:
  
  YYYY-MM-DDT22:53:59.956Z cpu27:2104032)uw.2104033 (39550) extraMin/extraFromParent: 1921/1921, sh.2104025 (39475) childEmin/eMinLimit: 25590/25600
YYYY-MM-DDT22:53:59.956Z cpu27:2104032)Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.2104025:python.2104033:uw.2104033
YYYY-MM-DDT22:53:59.958Z cpu27:2104032)WARNING: LinuxThread: 424: python: Error cloning thread: -28 (bad0081)
YYYY-MM-DDT22:53:59.960Z cpu27:2104032)Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.2104025:python.2104034:uw.2104034
YYYY-MM-DDT22:53:59.960Z cpu27:2104032)uw.2104034 (39559) extraMin/extraFromParent: 39/39, sh.2104025 (39475) childEmin/eMinLimit: 25581/25600
YYYY-MM-DDT22:53:59.960Z cpu27:2104032)Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.2104025:python.2104034:uw.2104034
YYYY-MM-DDT22:53:59.961Z cpu27:2104032)WARNING: LinuxThread: 424: python: Error cloning thread: -28 (bad0081)
YYYY-MM-DDT22:54:00.823Z cpu27:2104032)Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.2104025:python.2104035:uw.2104035
YYYY-MM-DDT22:54:00.823Z cpu27:2104032)uw.2104035 (39568) extraMin/extraFromParent: 512/512, sh.2104025 (39475) childEmin/eMinLimit: 25579/25600
YYYY-MM-DDT22:54:00.823Z cpu27:2104032)Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.2104025:python.2104035:uw.2104035


• In the /var/log/hostd.log file of the ESXi host, below events are seen:

YYYY-MM-DDT22:53:59.533Z info hostd[2100566] [Originator@6876 sub=Vimsvc.TaskManager opID=esxui-7618-94e7 user=root] Task Created : haTask-ha-host-vim.host.ImageConfigManager.fetchSoftwarePackages-450
YYYY-MM-DDT22:53:59.534Z info hostd[2100586] [Originator@6876 sub=SysCommandPosix opID=esxui-7618-94e7 user=root] ForkExec(/usr/bin/sh) 2104025

Environment

VMware vSphere 7.0.x

Cause

The issue occurs in vendor images with customized drivers where ImageConfigManager consumes more RAM than allocated. As a result, repeated failures of ImageConfigManager cause ESXi hosts to disconnect from vCenter Server.

Resolution

This issue is fixed in 7.0 U2C. Refer to VMware ESXi 7.0 Update 2c Release Notes.