ESXi host goes into not responding state frequently in vCenter and hostd crashes
search cancel

ESXi host goes into not responding state frequently in vCenter and hostd crashes

book

Article ID: 402079

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • ESXi hosts goes not responding in vCenter and recovers automatically few minutes later. 
  • In /var/run/log/hostd.log and /var/run/log/vmkernel.log you see the hostd crash 

hostd.log
YYYY-MM-DDTHH:MM:SSZ error hostd[15818064] [Originator@6876 sub=Libs opID=nsxDaVim-85-8884 user=dcui] [ConfigStore:227657205504:] [cs:2:1768466420]Failed to execute[BEGIN IMMEDIATE;](database is locked) rc=5
YYYY-MM-DDTHH:MM:SSZ error hostd[15818064] [Originator@6876 sub=Libs opID=nsxDaVim-85-8884 user=dcui] [ConfigStore:227657205504:] [cs:2:1768466420]Failed to start transaction rc = 1
YYYY-MM-DDTHH:MM:SSZ error hostd[15818064] [Originator@6876 sub=Libs opID=nsxDaVim-85-8884 user=dcui] [ConfigStore:227657205504:] [2000] Begin Transaction failed for: SetInstances
YYYY-MM-DDTHH:MM:SSZ info hostd[15818064] [Originator@6876 sub=Libs opID=nsxDaVim-85-8884 user=dcui] [ConfigStore:227657205504:] ConfigStoreException: [context]zKq7AVICAgAAAJOwXAEPY29uZmlnc3RvcmUAAEI7A2xpYmNvbmZpZ3N0b3JlLnNvAADLlQIAuosGAS7BaWxpYnZta2N0bC5zbwABQsJpAk7Cumhvc3RkAAOTxChsaWJ2bWFjb3JlLnNvAAMxxigCNTa2ApJ9tgKw5rkCyR+6AlMhuoQjphgBbGlidmltLXR5cGVzLnNvAALallI=[/context]
YYYY-MM-DDTHH:MM:SSZ- time the service was last started 2025-05-30T00:00:21.196Z, Section for VMware ESX, pid=23474679, version=7.0.3, build=22851731, option=Release

YYYY-MM-DDTHH:MM:SSZ info hostd[23474888] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 685 : /bin/hostd crashed (4 time(s) so far) and an encrypted core file using keyId vmware:key/fqid/<VMWARE-NULL>/kmip%2d80424147%2dbb21%2d46d5%2d8721%2d91b9f0cba958/645a36b7%2d92fc%2d11ec%2db698%2d4201ac1802c3 might have been created at /var/core/hostd-zdump.000. This might have caused connections to the host to be dropped.

vmkernel.log

YYYY-MM-DDTHH:MM:SSZ cpu35:23474299)osfs: OSFS_GetMountPointList:3726: mountPoints[0] inUse pid [    vsan], cid 5221830ad97de1dd-c8876afc1d3c36f8
YYYY-MM-DDTHH:MM:SSZ cpu67:15818064 opID=f2466c7d)World: 12077: VC opID nsxDaVim-85-8884 maps to vmkernel opID f2466c7d
YYYY-MM-DDTHH:MM:SSZ cpu67:15818064 opID=f2466c7d)User: 3220: hostd-worker: wantCoreDump:hostd-worker signal:6 exitCode:0 coredump:enabled
YYYY-MM-DDTHH:MM:SSZ cpu67:15818064 opID=f2466c7d)UserDump: 2635: hostd-worker: Dumping cartel 15818025 (from world 15818064) to file /var/core/hostd-zdump.000 ...

Environment

VMware vSphere ESXi 7.x

Cause

  • Hostd service crashed causing the host to go to not responding state. The service crashed due to db lock. ConfigStore transaction errors occur when one connection is waiting on DB lock, which is unavailable. It gives up after trying for 15 seconds causing the hostd to crash.

Resolution

  • The issue is fixed in ESXI version 8.0u3 and above.

Workaround: Currently there is no workaround.

Powered by Wolken Software