Virtual machine power-on task fails with an error "Failed to instantiate VMM bootstrap blob: Failure" or after bootup VNIC activation failed "Activate request failed for device ##"

Products

• VMware vSphere ESXi

Issue/Introduction

• When attempting to power on or vMotion a virtual machine (VM), the task fails with an error:

  This virtual machine failed to switch to 64 bit mode. This failure might be due to a lack of available memory on the host.
  
  

• VM may crash during vMotion.

• After successful vMotion and power on some of the VNICs activation failed with the error message in VM’s vmware.log 

  [YYYY-MM-DDTHH:MM:SS]  In(05) vcpu-6 - VMXNET3 user: failed to activate 'Ethernet4', status: 0xbad0001
  [YYYY-MM-DDTHH:MM:SS]  In(05) vcpu-6 - VMXNET3 user: Activate request failed for device ##.
  

• When reviewing /var/run/log/vmkernel.log of the host where the power on failed, or on the target host in case of a vMotion, the following entries are seen:

  [YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu38:18#######)Vmxnet3: 11994: Failed to map the rx data ring for rq [Queue no]
  [YYYY-MM-DDTHH:MM:SS]In(182) vmkernel: cpu126:18#######)Vmxnet3: 11575: Failed to map the tx ring for tq [Queue no ]
  
  We will see the following informational messages for DPDK deployed VM’s: 
  
  [YYYY-MM-DDTHH:MM:SS]vmkernel: cpu10:4#######)Vmxnet3: 459: Memory region is completely overlapped 400###
  [YYYY-MM-DDTHH:MM:SS]vmkernel: cpu51:4#######)Vmxnet3: 459: Memory region is completely overlapped 400###
  

• NSX Edge VMs are affected by this issue with unique symptoms. Sometimes after the boot up some of the edge services will not start.  NSX manager Configuration state shows as 'Pending' on NSX manager UI. 
  
  Following messages will be seen in /var/log/syslog on the affected NSX edge VM:

  [YYYY-MM-DDTHH:MM:SS]  [hostname]  NSX FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="INFO"] vmxnet3_dev_configure(): disable UPTv2
  [YYYY-MM-DDTHH:MM:SS]  [hostname]  NSX FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="INFO"] eth_vmxnet3_setup_capabilities(): UPTv2 disabled
  [YYYY-MM-DDTHH:MM:SS]  [hostname]   NSX FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="INFO"] eth_vmxnet3_setup_capabilities(): Dev 0000:####, vmxnet3 v8, UPT enabled: false, DCR0=0x00#######, used DCR=0x00#####, PTCR=0x80000000, used PTCR=0x00000000
  [YYYY-MM-DDTHH:MM:SS]  [hostname]   NSX FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="WARN"] vmxnet3_dev_start(): Device activation: UNSUCCESSFUL
  [YYYY-MM-DDTHH:MM:SS]  [hostname]  NSX FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="INFO"] net_vmxnet3 port 2 start failed: Invalid argument, retrying (2)
  

Environment

• VMware vSphere ESXi 8.0.x

Cause

This is caused by XMap fragmentation/ exhaustion for the VMXNet3 device

Resolution

This issue can be resolved by increasing the xmap memory size to 64 GByte. Follow the below steps:  

• • Run the following command on the affected host. This will reserve more space for xmap and allow more vms to power on / resolve the config state 'Pending' on Edge VM:

    # esxcli system settings kernel set -s "xmapGB" -v 64

  • Reboot the host to allow its kernel to pickup the change.