vNUMA is disabled if vCPU hotplug is enabled for a virtual machine on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

When selecting the option Enable CPU Hot Add in the virtual machines Edit Settings dialogue, or manually setting the option vcpu.hotadd = "true" in the VMX configuration file of a virtual machine, vNUMA is disabled for the VM. As a result, even if the virtual machine is created with more vCPUs than cores in one of the underlying hardware's NUMA nodes (or threads if numa.vcpu.preferHT = "true" is set), only one virtual node (VPD) will be presented to the VM and the memory interleaved over the multiple resulting NUMA clients (PPDs).

When reviewing the VMX log of the VM, /vmfs/volumes/<datastore>/<vm-folder>/vmware.log, you find the following entries:

<datetime> vmx| I125: numa: Hot add is enabled and vNUMA hot add is disabled, forcing UMA.

 

or

<datetime> vmx - numa: Hot-add is enabled and vNUMA hot-add is disabled, forcing UMA.

Environment

VMware vSphere ESXi 6.x

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Resolution

If you do not plan to use vCPU hotplug, make sure to disable it, and instead add the maximum number of vCPUs that might be needed by the workload. In rare cases where vCPU hot plug is required, try to limit the maximum number of vCPUs to stay below the amount of cores in the physical NUMA node, so that the virtual machine can be scheduled in a single NUMA node.

Steps to Enable/ Disable CPU Hot Add:

1. Login to vCenter UI
2. Locate the VM and Power it off.
3. Right click on the Virtual Machine (centos8) > Edit Settings > Virtual Hardware > expand CPU > CPU Hot Plug.
4. To disable CPU hot Plug: Uncheck the box "Enable CPU Hot Add".
5. To enable CPU hot Plug: Check the box "Enable CPU Hot Add". 

Additional Information

Using Virtual NUMA (VMware Docs)

https://knowledge.broadcom.com/external/article?articleNumber=340275