Severe CPU consumption spikes on an ESXi host, which are traced back to the resource demands of a single virtual machine (running AV)

Products

• VMware vSphere ESXi

Issue/Introduction

Performance monitoring confirms that a single 24-vCPU virtual machine drives excessive host CPU usage, which degrades performance even though the host has adequate overall compute capacity.

 

Environment

ESXi 8.0U3

Cause

The host CPU spikes are caused by NUMA Thrashing (Non-Uniform Memory Access thrashing) due to a misalignment between the virtual machine's vCPU configuration and the physical host's CPU architecture. The underlying physical ESXi host contains dual 18-core processors. Because the virtual machine is assigned 24 vCPUs, its compute footprint is too wide to fit onto a single physical CPU (NUMA node). Due to an improper virtual hardware configuration (and potentially Sub-NUMA Clustering/SNC being enabled), the guest operating system (Windows) is blind to the underlying physical CPU boundary

Resolution

To eliminate the thrashing and align the virtual architecture with the physical hardware configure the VM using the following strict parameters: ( This is presented as an example.)

• vCPU Count: 24

• Cores per Socket: 1 (Resulting in 24 Virtual Sockets). Note: Leaving this at 1 is the VMware best practice to allow the hypervisor to automatically calculate the optimal vNUMA topology.

• Advanced Settings: Ensure no advanced vmx settings explicitly override the vNUMA engine.

• Host BIOS: Ensure Sub-NUMA Clustering (SNC) is disabled at the physical server level.

What happens under the hood: Because the 24 vCPUs cross the physical 18-core boundary, vSphere's vNUMA (Virtual NUMA) engine activates. ESXi cleanly splits the VM exactly in half: it places 12 vCPUs and 50% of the RAM on Physical CPU 1, and 12 vCPUs and 50% of the RAM on Physical CPU 2. vSphere then presents this Virtual NUMA topology directly to the Windows OS.

When Windows boots, it recognizes a dual-processor system with 12 cores each. Fully aware of this 12/12 split, the Windows OS Memory Manager takes over. When the AV engine spins up a thread on Node 1, Windows intentionally forces that thread to read RAM physically located on Node 1. By relying on vNUMA, the VM gains the brute force of high thread counts (24 vCPUs) alongside the extreme efficiency of localized memory access.