ESXi 6.7 host shows not responding in vCenter Server and log report Heap vfat already at its maximum size

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi host shows as not-responding in vCenter Server Web Client and HTML5 Client.
• Cannot connect directly to host.
• Cannot SSH to host.
• When in DCUI console to host, pressing F2 does nothing

The vmkwarning.log has entries similar to the following vfat heap warnings:

2019-01-14T09:40:18.322Z cpu99:2098151)WARNING: Heap: 4079: Heap_Align(vfat, 131072/131072 bytes, 8 align) failed. caller: 0x41803302dbd4
2019-01-14T10:08:04.810Z cpu38:2097982)WARNING: Heap: 4079: Heap_Align(vfat, 131072/131072 bytes, 8 align) failed. caller: 0x41803302dbd4
2019-01-14T10:11:17.993Z cpu64:2098475)WARNING: vmkusb: udev 0x4308f1222118, endpoint 0x81: invalid state 5: Failure

The syslog.log has "out of memory" entries:

2019-01-13T13:16:11Z Unknown: out of memory [2859336]

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.

Cause

Due to a timing issue in the VMkernel, buffers might not be flushed, and the heap gets exhausted. As a result, services such as hostd, vpxa and vmsyslogd might not be able to write logs on the ESXi host.

Resolution

This issue is resolved in ESXi 6.7 U2 release. For more information see release notes

To download go to Customer Connect

Workaround:
There is no known workaround for this issue.

Additional Information

Impact/Risks:
Requires ESXi host reboot.