Power on VM Fails with the Errror "Failed to instantiate VMM bootstrap blob: Failure"

Products

• VMware vCenter Server 8.0

Issue/Introduction

Powering on VM fails with the error "Module MonitorLoop power on failed"

Error stack:
 Module Monitor power on failed.
 This virtual machine failed to switch to 64 bit mode. This failure might be due to a lack of available memory on the host.

vmkernel logs:
[YYYY-MM-DDTHH:MM:SS] vmkernel: cpu64:2417769)World: vm 2417772: 7244: Starting world vmm0:Name_of_the_vm of type 8
[YYYY-MM-DDTHH:MM:SS] vmkwarning: cpu64:2417769)WARNING: World64: 1112: Failed to instantiate VMM bootstrap blob: Failure

[YYYY-MM-DDTHH:MM:SS]vmkernel: cpu10:4322487)Vmxnet3: 459: Memory region is completely overlapped 4004dc2
[YYYY-MM-DDTHH:MM:SS]vmkernel: cpu51:4322965)Vmxnet3: 459: Memory region is completely overlapped 4004dcd

Environment

vSphere

Cause

This is Due to vmxnet3 devices cause XMap fragmentation/exhaustion

Resolution

Vmxnet3 memory regions is a feature will allow certain performance centric vms/vnics such as dpdk vms to premap some memory for better performance.

Following workarounds can be done to mitigate/resolve the issue.

1. Increase xmap size to 64GB using below boot config option. This will require reboot of the host. This will reserve more space for xmap and allow more vms to power on.
    esxcli system settings kernel set -s "xmapGB" -v 64

OR

2. We can disable memory regions for the vms using below option. This will require vms to be power cycled. This will not allow memory regions to be configured for the vms, so xmap memory will not be exhausted. We don't need to use option (1) if we do this.
   esxcfg-advcfg -s 0 /Net/Vmxnet3MemoryRegions

OR

3. We can limit the amount of memory requested per device. This will allow vms to use memory regions as long as requested memory is within the bound i.e. 64 MB in below setting. This will require reboot of the vms. Also, this can still lead to xmap exhaustion if there are way too many vms all requesting such memory feature. This can be don along with option 1.

esxcfg-advcfg -s 64 /Net/NetMaxMemRegPerDevice