ESXi Hosts go into 'not responding' state intermittently on GPU hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• hostd service is crashing on the host causing the host to go into a not responding state and generates a hostd dump on the host
• Hostd dump output: 
  

  #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
  #1  0x0000008335f2fbdf in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
  #2  0x0000008335ee56d2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
  #3  0x0000008335ed065c in __GI_abort () at abort.c:79
  #4  0x000000832ecb2a4f in Vmacore::System::SignalTerminateHandler (info=info@entry=0x0, ctx=ctx@entry=0x0) at bora/vim/lib/vmacore/posix/defSigHandlers.cpp:62
  #5  0x000000832ecb2aa2 in Vmacore::System::TerminateHandler () at bora/vim/lib/vmacore/posix/defSigHandlers.cpp:78
  #6  0x000000832b2fb156 in __cxxabiv1::__terminate (handler=<optimized out>)
      at /build/mts/release/bora-24499230/cayman_gcc/gcc/src/libstdc++-v3/libsupc++/eh_terminate.cc:48
  #7  0x000000832b2fa259 in __cxa_call_terminate (ue_header=ue_header@entry=0x82ebdc7ea0)
      at /build/mts/release/bora-24499230/cayman_gcc/gcc/src/libstdc++-v3/libsupc++/eh_call.cc:54
  #8  0x000000832b2fab84 in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=6, exception_class=5138137972254386944, ue_header=0x82ebdc7ea0,
      context=<optimized out>) at /build/mts/release/bora-24499230/cayman_gcc/gcc/src/libstdc++-v3/libsupc++/eh_personality.cc:677
  #9  0x0000008335e99823 in _Unwind_RaiseException_Phase2 (exc=0x82ebdc7ea0, context=0x833907ff70, frames_p=0x833907fe78)
      at /build/mts/release/bora-24499230/cayman_gcc/gcc/src/libgcc/unwind.inc:64
  #10 0x0000008335e9a086 in _Unwind_Resume (exc=exc@entry=0x82ebdc7ea0) at /build/mts/release/bora-24499230/cayman_gcc/gcc/src/libgcc/unwind.inc:241
  #11 0x0000008333d38dc8 in Vim::ConfigStore::Util::GetMappings (mappings=..., path=..., createObjects=<optimized out>) at bora/vim/lib/vimConfigStore/util.cpp:293
  #12 0x0000008333d5f72e in Vim::ConfigStore::Util::GetViewHelper (
      mappings=std::shared_ptr<std::vector<Vim::ConfigStore::Impl::PathRootObject, std::allocator<Vim::ConfigStore::Impl::PathRootObject> >> (use count 12, weak count 0) = {...}, settings=..., path=..., defaultsView=..., key=..., view=...)
      at external/cayman_esx_toolchain_gcc9/usr/bin/../lib/gcc/x86_64-vmk-linux-gnu/9.3.0/../../../../x86_64-vmk-linux-gnu/include/c++/9.3.0/bits/shared_ptr_base.h:1020
  #13 0x0000008333d3b3f6 in Vim::ConfigStore::Impl::ConfigStoreConfigImpl::GetView (this=0x82eb9f9f00, key="plugins/hostsvc", view=...)
      at bora/vim/lib/vimConfigStore/impl.cpp:456
  #14 0x00000082ea2a3c08 in SynchronizedConfig::GetView (this=this@entry=0x82eba2c520, key=..., out=...) at bora/vim/hostd/main/hostdAppInt.h:211
  #15 0x00000082ea02e5b5 in Hostsvc::ActiveDirectoryAuthenticationImpl::GetMaintenanceInterval (this=this@entry=0x8337e24450)
      at bora/vim/hostd/hostsvc/impl/ActiveDirectoryAuthenticationImpl.cpp:1413
  #16 0x00000082ea030155 in Hostsvc::ActiveDirectoryAuthenticationImpl::DoMaintenance (this=0x8337e24450)
      at bora/vim/hostd/hostsvc/impl/ActiveDirectoryAuthenticationImpl.cpp:1624

• Via SSH session into the ESXi host the following has an output of 1000+ hostd leaked character devices: 
  lsof | grep vmgfx | sort | uniq -w10 -c | grep hostd
  
  • Example output:
     3253 23940381    hostd                 CHAR                     1000   /dev/char/vmkdriver/vmgfx6

Environment

VMware vSphere 9.0.x

Resolution

This should be resolved in VCF 9.1. 

Workaround: 
Disable the GPU stats: 

1. SSH into the ESXi host(s) and run the following commands: 
   configstorecli config current set -c esx -g services -k hostd -p plugins/statssvc/collect_gpu_stats -v false
2. To check the status is false run the following: 
   configstorecli config current get -c esx -g services -k hostd
3. Restart the hostd service for the change to take affect: 
   /etc/init.d/hostd restart

 

Note: Disabling the GPU stats will not affect DRS as GPU stats are not used by DRS.