This issue occurs when ESXi hosts access the boot devices from other ESXi hosts.
In a supported configuration, there can only be a maximum of 4 vfat partitions on a single boot device (boot LUN). The vfat heap size is exhausted if too many vfat partitions from multiple ESX boot partitions are presented to the host.
For more information, see the Boot from Fibre Channel SAN Requirements and Considerations section in the vSphere Storage Guide.
Resolution
To resolve this issue, ensure that only a single boot LUN from the SAN Array is presented to each ESXi host. This should be the boot LUN for that host only.