Error: "High latency and performance issues" as vNUMA is disabled if vCPU hotplug is enabled for a virtual machine on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• When selecting the option Enable CPU Hot Add in the virtual machines Edit Settings dialogue, or manually setting the option vcpu.hotadd = "true" in the VMX configuration file of a virtual machine, vNUMA is disabled for the VM. As a result, even if the virtual machine is created with more vCPUs than cores in one of the underlying hardware's NUMA nodes (or threads if numa.vcpu.preferHT = "true" is set), only one virtual node (VPD) will be presented to the VM and the memory interleaved over the multiple resulting NUMA clients (PPDs).
  
  When reviewing the VMX log of the VM, /vmfs/volumes/<datastore>/<vm-folder>/vmware.log, you find the following entries:

  <datetime> vmx| I125: numa: Hot add is enabled and vNUMA hot add is disabled, forcing UMA.

• Or

  <datetime> vmx - numa: Hot-add is enabled and vNUMA hot-add is disabled, forcing UMA.

Environment

VMware vSphere ESXi 8.X

VMware vSphere ESXi 7.X

Cause

• Hot Add enabled
• NUMA topology presented to VM reflects the host, resulting in better performance for the VM

Resolution

1. If you do not plan to use vCPU hotplug, make sure to disable it, and instead add the maximum number of vCPUs that might be needed by the workload. In rare cases where vCPU hot plug is required, try to limit the maximum number of vCPUs to stay below the amount of cores in the physical NUMA node, so that the virtual machine can be scheduled in a single NUMA node.
2. Steps to Enable/ Disable CPU Hot Add:
   1. Login to vCenter UI
   2. Locate the VM and Power it off.
   3. Right click on the Virtual Machine (centos8) > Edit Settings > Virtual Hardware > expand CPU > CPU Hot Plug.
   4. To disable CPU hot Plug: Uncheck the box "Enable CPU Hot Add".
   5. To enable CPU hot Plug: Check the box "Enable CPU Hot Add". 

Additional Information

• Using Virtual NUMA (VMware Docs)
• Enabling vCPU HotAdd creates fake NUMA nodes on Windows