Multiple VMs lost network connection - failed to activate 'Ethernet0', status: 0xbad0014

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

• Virtual Machines are randomly going into hung state after successfully migrating to another ESXI Host
• After successful completion of VM migration vmxnet3 driver will not initialize.
• The VM is not reachable over the network but in the console, it appears to be running.
• Can't see any traffic on SwitchPort packet capture 
• vmware.log
  

  In(05) vcpu-0 - VMXNET3 user: failed to activate 'Ethernet0', status: 0xbad0014
  
  In(05) vcpu-0 - VMXNET3 user: Activate request failed for device 0.

Environment

VMware ESXi 8.0.X

Cause

• The vNIC activation failure "VMXNET3 user: failed to activate 'Ethernet0', status: 0xbad0014"  indicates "Out of memory"
• This issue happens when available memory on the system is in a very fragmented state where the OS is not able to find/allocate contiguous large memory blocks. 

Resolution

This issue is resolved in ESX 8.0.3 P06(24859861) and newer versions, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Restarting the VM resolves the issue by clearing system memory and resetting the VMXNET3 driver configurations.

Additional Information

Build numbers and versions of VMware ESXi/ESX