ESXi Host enters Not Responding state in vCenter due to hostd Crash with error "fips module entering error state"
search cancel

ESXi Host enters Not Responding state in vCenter due to hostd Crash with error "fips module entering error state"

book

Article ID: 424063

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

- The hostd service crashes repeatedly with hostd core dumps generated after upgrading ESXi from version 7.0 to 8.0 on Lenovo hardware 

- We could see the below entries in var/run/log/host.log:

YYYY-MM-DDTHH:MM:SS.SSSZ Er(163) Hostd[####]: [Originator@6876 sub=Libs] CryptoRandom_GetBytes failed
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: [Originator@6876 sub=Default] 
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> 
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> Panic: UUID generation failure
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> Backtrace:
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> [backtrace begin] product: VMware ESX, version: 8.0.3, build: build-24859861, tag: hostd, cpu: x86_64, os: esx, buildType: release
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[00] libvmacore.so[0x####]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[01] libvmacore.so[0x####]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[02] libvmacore.so[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[03] libvmacore.so[0x####]: Vmacore::PanicExit(char const*)
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[04] libvmacore.so[0x####]: Vmacore::StringUtil::GenerateUuid()
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[05] libvmacore.so[0x####]: Vmacore::CreateSession(long, Vmacore::Ref<Vmacore::Session>&)
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[06] libvmacore.so[0x####]: Vmacore::Http::HttpSessionResolver::CreateAppSession(long) const
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[07] libvmacore.so[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[08] libvmacore.so[0x####]: Vmacore::Http::ResolveSession(Vmacore::Http::SessionResolver const&, Vmacore::Ref<Vmacore::Session>&)
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[09] libvmacore.so[0x####]: Vmacore::Http::ResolveSession(Vmacore::Http::Request*, std::string const&, Vmacore::Http::Response*, Vmacore::Ref<Vmacore::Session>&)
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[10] libvmomi.so[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[11] libvmacore.so[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[12] libvmacore.so[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[13] libvmacore.so[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[14] libvmacore.so[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[15] libpthread.so.0[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[16] libc.so.6[0x####]
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> backtrace[17] (no module)
YYYY-MM-DDTHH:MM:SS.SSSZ Cr(162) Hostd[####]: --> [backtrace end]
.
.
YYYY-MM-DDTHH:MM:SS.SSSZ Er(163) Hostd[####]: [Originator@6876 sub=AdapterServer opID=agw-#### sid=#### user=vpxuser:VSPHERE.LOCAL\vpxd-extension-####] AdapterServer caught exception; <<####, <TCP '127.0.0.1 : ####'>, <TCP '127.0.0.1 : ####'>>, vsanSystem, vim.host.VsanSystem.fetchVsanSharedSecret, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x000000e884ae8928]>, N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:1C8000E0:Provider routines::fips module entering error state)
.
.
YYYY-MM-DDTHH:MM:SS.SSSZ Er(163) Hostd[####]: [Originator@6876 sub=Libs] CryptoRandom_GetBytes failed


- Analysis of the hostd core dumps revealed the following back trace entries, clearly indicating a "UUID generation failure":

#0  __GI_raise (sig=sig@entry=6) at ####
#1  0x#### in __GI_abort () at abort.c:####
#2  0x#### in Vmacore::PanicExit (msg=msg@entry=0x#### "UUID generation failure") at ####
#3  0x#### in Vmacore::StringUtil::GenerateUuid () at ####


Environment

VMware ESXi 7.0.x
VMware ESXi 8.0.x

Cause

The hostd service may crash intermittently due to an OpenSSL FIPS self-test failure ('FIPS module entering error state'), typically triggered by faulty hardware

Resolution

Please contact the respective hardware vendor to investigate further.

Additional Information

fips module entering error state エラーによる hostd クラッシュが原因で vCenter 上の ESXi ホストが「応答なし」状態になる

Powered by Wolken Software