To resolve the hostd issue.
Symptoms:
yyyy-mm-ddTxx:xx:xx.xxxZ error hostd[309C2B70] [Originator@6876 sub=UW Memory checker] Current value 1288636 exceeds hard limit 1286881. Shutting down process.
yyyy-mm-ddTxx:xx:xx.xxxZ panic hostd[309C2B70] [Originator@6876 sub=Default]
-->
--> Panic: Memory exceeds hard limit. Panic
--> Backtrace:
--> [backtrace begin] product: VMware ESX, version: 6.5.0, build: build-6765664, tag: hostd, cpu: x86, os: esx, buildType: release
--->
Line 16733: yyyy-mm-ddTxx:xx:xx.xxxZ info hostd[2C297B70] [Originator@6876 sub=Libs] ToolsISO: Selected Tools ISO 'windows.iso' for 'windows7srv-64' guest.
Line 16744: yyyy-mm-ddTxx:xx:xx.xxxZ info hostd[2C297B70] [Originator@6876 sub=Libs] ToolsISO: Error hash mismatch at offset 0x8000, expected hash [PbIrjdgFyu824nbRIbnFi8+DNkHIO06vZK/AIFcymIo=], but found [kBzqaLCzmb61r+AtU/ft/+kgRiR8V9OgDtOq6QQlQrI=]
Line 16745: yyyy-mm-ddTxx:xx:xx.xxxZ info hostd[2C297B70] [Originator@6876 sub=Libs] ToolsISO: Invalid data at offset 8000 (bytes 2048)
Line 16746: yyyy-mm-ddTxx:xx:xx.xxxZ info hostd[2C297B70] [Originator@6876 sub=Libs] Failed to open Tools ISO image.
Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
The hash check fails for VMware Tools ISO and the hostd process fails to open/extract the ISO file. Due to this failure, the allocated hash info is not being cleaned up efficiently, leading to memory leak and the eventual hostd crash.
VMware vSphere ESXi 6.5 P05, and VMware vSphere ESXi 6.7 P02 has the fix for the issue.
Workaround:
To workaround this issue follow the below steps.
Copy the /productLocker/vmtools/ folder and its contents from a working host running the same ESXi version/build and replace them on the affected host.
1. Enter the ESXi host in Maintenance Mode
2. Enable SSH on both the "good" (for e.g. esxi01 ) and the "affected" (esxi02) host.
3. Login as "root" user
4. scp the /productLocker/vmtools/ folder and contents from the "good" host to the "problem" host
$ [esxi01:~]$ scp -rf /productLocker/vmtools/ esxi02:/productLocker/
Note: This will overwrite the vmtools folder on the problem host, hence you could backup this folder locally if you want but it is not mandatory for the purpose of this activity.
5. Verify the MD5SUM of the vmtools folder contents on the good and bad host. These should match after copying.
For e.g. md5sum /productLocker/vmtools/windows.iso
1f13ee2eb9dfb4eab070c241a1d71cf4 /productLocker/vmtools/windows.iso
Note: Run the md5sum command on all the files under the vmtools folder on both hosts
5. Restart Management Agents
services.sh restart
6. Exit Maintenance Mode