ESXi Host enters Not Responding state in vCenter due to hostd Crash with error "fips module entering error state"
search cancel

ESXi Host enters Not Responding state in vCenter due to hostd Crash with error "fips module entering error state"

book

Article ID: 424063

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

- The hostd service crashes repeatedly with hostd core dumps generated after upgrading ESXi from version 7.0 to 8.0 on Lenovo hardware 

- We could see the below entries in var/run/log/host.log:

YYYY-MM-DDTHH:MM:SS.083Z Er(163) Hostd[2099476]: [Originator@6876 sub=Libs] CryptoRandom_GetBytes failed
YYYY-MM-DDTHH:MM:SS.084Z Cr(162) Hostd[2099476]: [Originator@6876 sub=Default] 
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> 
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> Panic: UUID generation failure
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> Backtrace:
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> [backtrace begin] product: VMware ESX, version: 8.0.3, build: build-24859861, tag: hostd, cpu: x86_64, os: esx, buildType: release
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> backtrace[00] libvmacore.so[0x003BFDE2]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> backtrace[01] libvmacore.so[0x003D7487]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> backtrace[02] libvmacore.so[0x0051E4DD]
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> backtrace[03] libvmacore.so[0x0051E62B]: Vmacore::PanicExit(char const*)
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> backtrace[04] libvmacore.so[0x0047EC3C]: Vmacore::StringUtil::GenerateUuid()
YYYY-MM-DDTHH:MM:SS.093Z Cr(162) Hostd[2099443]: --> backtrace[05] libvmacore.so[0x00499F1B]: Vmacore::CreateSession(long, Vmacore::Ref<Vmacore::Session>&)
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[06] libvmacore.so[0x003FACAE]: Vmacore::Http::HttpSessionResolver::CreateAppSession(long) const
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[07] libvmacore.so[0x003FC945]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[08] libvmacore.so[0x003FF86E]: Vmacore::Http::ResolveSession(Vmacore::Http::SessionResolver const&, Vmacore::Ref<Vmacore::Session>&)
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[09] libvmacore.so[0x003FFEE1]: Vmacore::Http::ResolveSession(Vmacore::Http::Request*, std::string const&, Vmacore::Http::Response*, Vmacore::Ref<Vmacore::Session>&)
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[10] libvmomi.so[0x0020C92D]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[11] libvmacore.so[0x0043BF6A]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[12] libvmacore.so[0x002CDB1E]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[13] libvmacore.so[0x002CFFE0]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[14] libvmacore.so[0x0052504B]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[15] libpthread.so.0[0x00007852]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[16] libc.so.6[0x000F523F]
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> backtrace[17] (no module)
YYYY-MM-DDTHH:MM:SS.094Z Cr(162) Hostd[2099443]: --> [backtrace end]
.
.
YYYY-MM-DDTHH:MM:SS.042Z Er(163) Hostd[2099489]: [Originator@6876 sub=AdapterServer opID=agw-*******-****-**-**-**** sid=52545f53 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-********-****-****-aaac-be**********] AdapterServer caught exception; <<********-****-****-****-efbd******b0, <TCP '127.0.0.1 : ****'>, <TCP '127.0.0.1 : *****'>>, vsanSystem, vim.host.VsanSystem.fetchVsanSharedSecret, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x000000e884ae8928]>, N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:1C8000E0:Provider routines::fips module entering error state)
.
.
YYYY-MM-DDTHH:MM:SS.083Z Er(163) Hostd[2099476]: [Originator@6876 sub=Libs] CryptoRandom_GetBytes failed


- Analysis of the hostd core dumps revealed the following back trace entries, clearly indicating a "UUID generation failure":

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000000894c72ae04 in __GI_abort () at abort.c:79
#2  0x00000089441e358a in Vmacore::PanicExit (msg=msg@entry=0x8944643fa6 "UUID generation failure") at bora/vim/lib/vmacore/posix/panic.cpp:201
#3  0x000000894441fc3c in Vmacore::StringUtil::GenerateUuid () at bora/vim/lib/vmacore/main/lib.cpp:1494


Environment

VMware ESXi 7.0.x
VMware ESXi 8.0.x

Cause

The hostd service may crash intermittently due to an OpenSSL FIPS self-test failure ('FIPS module entering error state'), typically triggered by faulty hardware

Resolution

Please contact the respective hardware vendor to investigate further.

Powered by Wolken Software