ESXi 7.0 host goes not responding in vCenter and hostd.log reports "SetVigorNotificationTime: Not connected"

Products

• VMware vSphere ESX 7.x

Issue/Introduction

• ESXi 7.0 host goes not responding in vCenter and recovers automatically few minutes later showing signs of instability
• Datastores will show very high IO usage. When reviewed with esxtop batch data the usage is coming from the hostd process.
• ESXi host is seeing very high CPU consumption
• "esxtop' shows hostd process consuming high %RDY
• Restarting hostd services brings the host into a normal state but will show high CPU usage again after sometime
• In ESXi - /var/log/hostd.log, the hostd process crashes with a panic
  
  YYYY-MM-DDTHH:MM:SS.583Z error hostd[2101222] [Originator@6876 sub=UW Memory checker] Current value 2106136 exceeds hard limit 2105548. Shutting down process.
YYYY-MM-DDTHH:MM:SS.583Z panic hostd[2101222] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware ESX, version: 7.0.3, build: build-20842708, tag: hostd, cpu: x86_64, os: esx, buildType: release
--> backtrace[00] libvmacore.so[0x001D6812]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001BD35D]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x003E64AB]
--> backtrace[03] libvmacore.so[0x003E6584]: Vmacore::PanicExit(char const*)
--> backtrace[04] libvmacore.so[0x001C5831]: Vmacore::System::ResourceChecker::DoCheck()
--> backtrace[05] libvmacore.so[0x0037EC70]
--> backtrace[06] libvmacore.so[0x002DC7CC]
--> backtrace[07] libvmacore.so[0x002E0354]
--> backtrace[08] libvmacore.so[0x003F1102]
--> backtrace[09] libpthread.so.0[0x00007D3B]
--> backtrace[10] libc.so.6[0x000ED16D]
--> backtrace[11] (no module)
--> [backtrace end]

Note: "Memory exceeds hard limit" messages may not be noticed in every scenarios.

• In ESXi - /var/log/hostd.log, there is many messages of:
  
  YYYY-MM-DDTHH:MM:SS.112Z warning hostd[2342313] [Originator@6876 sub=VigorStatsProvider(000000fd581cd360).GuestStats(43328) opID=lro-<OP_ID>] SetVigorNotificationTime: Not connected

Environment

VMware vSphere ESXi 7.x

Cause

This issue is caused by race conditions between the power-off operation on the VM and the actual moment when the Vigor callbacks are invoked. As a result, the infinite loop can make the hostd process to consume memory to the hard limit.

Resolution

This issue is fixed in VMware ESXi 7.0 Update 3o; refer to the ESXi 7.0U3o release notes for more information. To download, go to Broadcom Support page.