Troubleshooting an Exception: Assert Failed error

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

When the hostd process encountered unexpected data in a file structure it was asserting.

Handling of this type of situation has been improved in the code.

Symptoms:

• Unable to connect the ESX Server host in the VirtualCenter
• Cannot connect to the host directly with VI Client either
• Hostd.log file contains an assertion failure similar to:
  
  Exception: Assert Failed: "statId != StatsStore::InvalidStatId" @ /build/mts/release/bora-68276/bora/vim/hostd/statssvc/statsDepot.cpp:548
  [2007-12-19 21:07:00.282 'App' 4158585744 error] Backtrace:
  [00] eip 0x66bdf8d Vmacore::System::BacktracePosix::BacktracePosix()
  [01] eip 0x6528e3b Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
  [02] eip 0x64bc958 Vmacore::Throwable::Throwable(std::string const&)
  [03] eip 0x64bc9ce Vmacore::PanicExit(std::string const&)
  [04] eip 0x64bcc2b Vmacore::RunTimeFailure(char const*, char const*, char const*, int, int)
  [05] eip 0x897a652 /usr/lib/vmware/bin/vmware-hostd [0x897a652]
  [06] eip 0x89425fd /usr/lib/vmware/bin/vmware-hostd [0x89425fd]
  [07] eip 0x89428d2 /usr/lib/vmware/bin/vmware-hostd [0x89428d2]
  [08] eip 0x66caa00 Vmacore::System::ThreadPoolPosix::RunWorkerThread()
  [09] eip 0x66c36ce Vmacore::System::ThreadPosixPrivate::ThreadBegin(void*)
  [10] eip 0x52a2db /lib/libpthread.so.0 [0x52a2db]
  [11] eip 0x46f14e clone

Environment

VMware ESX Server 2.5.x
VMware ESX Server 3.5.x

Resolution

Note: Before you begin, refer to Restarting the Management agents on an ESX Server (1003490) for important information on restarting the mgmt-vmware service.

To work around this problem:

1. Log in as root to the ESX Server with SSH.
2. Remove the entire /var/lib/vmware/hostd/stats directory with the following command:
   
   rm -fr /var/lib/vmware/hostd/stats
   
   
3. Restart the management agents. For more information, see Restarting the Management agents on an ESX Server (1003490).