Enabling vCPU HotAdd creates fake NUMA nodes on Windows

Products

• VMware vSphere ESXi

Issue/Introduction

When selecting "Enable CPU Hot Add" for any Virtual Machine, vNUMA is disabled. When listing NUMA information within a Windows guest with a variety of tools, you might see a second NUMA node (or more). This node is displayed:

• Without any associated CPUs or Memory for VMs with less than or exactly 64 vCPUs (not listed in e.g. Task Manager)
• With associated CPUs, but without any memory, for VMs with more than 64 vCPUs

Environment

• VMware vSphere ESXi 8.0.

Cause

Enabling vCPU Hot Add configures the virtual hardware for its supported and potentially hot plugged vCPU maximum, this depends on the selected Guest OS, ESXi version, Virtual Machine Compatibly (hardware version) and whether the virtual machine was powered on with more or less than 128 vCPUs.

For most Guest OS and Virtual Machine Compatibly selections and vCPUs below an initial configured amount of 128, vCPUs might be hot added up to 128 total vCPUs.

For some versions of Windows, when its Processor Group maximum of 64 is exceeded even with not yet live resources, Windows will create additional "fake" nodes to accommodate those potential vCPUs. This behaviour has changed in a recent versions of Windows, refer to Microsoft's NUMA Support documentation for details.

Resolution

As per Microsoft, this is expected behaviour before Windows 10 Build 20348

Workaround:

You can limit the maximum amount of vCPUs that can be hot added to be equal or below the Windows Processor Group maximum of 64 by configuring the following Virtual Machine advance configuration parameter and value:

cpuid.maxVCPUs = "integer value up to 64"

Additional Information

• Enable CPU Hot Add
• Error: "High latency and performance issues" as vNUMA is disabled if vCPU hotplug is enabled for a virtual machine on ESXi

Impact/Risks: Especially for Virtual Machines with more than 64 vCPUs, applications might make sub-optimal placement and scheduling decisions as the underlying virtual machine's memory, if wide, is interleaved.

Best Practice Recommendation: Disabling CPU Hot-Add is the preferred configuration for Windows Server guests to prevent the creation of "fake" NUMA nodes. Enabling this feature can lead to unpredictable application performance, increased latency, and potential kernel deadlocks under high load. We strongly recommend configuring the desired vCPU count in a powered-off state to ensure an optimized, static vNUMA topology.