Virtual machine power-on task fails with an error "Failed to instantiate VMM bootstrap blob: Failure"

Products

• VMware vSphere ESXi

Issue/Introduction

• When attempting to power on or vMotion a virtual machine (VM), the task fails with an error:

  Module MonitorLoop power on failed.
  This virtual machine failed to switch to 64 bit mode. This failure might be due to a lack of available memory on the host.

• VM may crash during vMotion
• when reviewing /var/run/log/vmkernel.log of the host where the power on failed, or on the target host in case of a vMotion, the following entries are seen:

  [YYYY-MM-DDTHH:MM:SS] vmkernel: cpu64:2417769)World: vm 2417772: 7244: Starting world vmm0:<Name_of_the_vm> of type 8
  [YYYY-MM-DDTHH:MM:SS] vmkwarning: cpu64:2417769)WARNING: World64: 1112: Failed to instantiate VMM bootstrap blob: Failure
  
  [YYYY-MM-DDTHH:MM:SS]vmkernel: cpu10:4322487)Vmxnet3: 459: Memory region is completely overlapped 4004dc2
  [YYYY-MM-DDTHH:MM:SS]vmkernel: cpu51:4322965)Vmxnet3: 459: Memory region is completely overlapped 4004dcd

Environment

VMware vSphere ESXi 8.0.x

Cause

This is caused by XMap fragmentation/exhaustion for the VMXNet3 device

Resolution

There are multiple ways to solve this issue:

1. By increasing the xmap memory size to 64 GByte

• • run the following command on the affected host. This will reserve more space for xmap and allow more vms to power on:

    # esxcli system settings kernel set -s "xmapGB" -v 64

  • reboot the host to allow its kernel to pickup the change.

 

2. By disabling the memory regions feature fore the VMs

• • This option should be used on its own, without any of the other 2 being set.
  • To implement this option, run the following command. This will not allow memory regions to be configured for the VMs, thus preventing the xmap memory from being exhausted:

    # esxcfg-advcfg -s 0 /Net/Vmxnet3MemoryRegions

  • Following this command, power-cycle the virtual machines currently running on the host, so they no longer use memory regions.

 

3. By Limiting the amount of memory that can be requested by the VMs for vmxnet3 memory regions

• • This option can either be used on its own, or it can be applied in addition to option 1.
  • To limit the amount each VM can request, run the following command. The value is in megabyte, e.g. the example will limit each VM to 64 MB:

    # esxcfg-advcfg -s 64 /Net/NetMaxMemRegPerDevice

  • To apply the change, power-cycle the virtual machines.

 

Note: Options 1 and/or 3 will not completely remove the risk of xmap exhaustion. If the number of virtual machines running on the host that use memory regions gets too large, the issue might still occur. If this happens, you can either select option 2 instead or reduce the number of VMs by spreading them out across other hosts.

Additional Information

The VMNet3 memory regions feature was implemented to allow performance-centric VMs/VNICs such as DPDK VMs to pre-map some memory for better performance.