High CPU utilization in vCenter with excessive NUMA migrations on ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

VMware vSphere ESXi. Customers observe a significant discrepancy between high CPU utilization reported at the vCenter/Host level and low CPU utilization reported within the Guest OS. Primary symptoms include excessive vCPU migrations (localityMig) occurring at high frequencies (e.g., every 5 seconds). Key identifiers include high NMIG counts and N%L values in ESXTOP.

Environment

 VMware ESXi Version: 8.0.3

Cause

NUMA Action Affinity (LocalityWeightActionAffinity) continuously forces vCPU migrations to co-locate virtual machine compute threads with physical NIC polling worlds (pollWorlds) during NUMA node load imbalances.

Resolution

To resolve the excessive migrations and associated CPU overhead, disable NUMA Action Affinity by following these steps:

1. Log into the ESXi host via SSH or the vSphere Client.
2. Navigate to the Advanced System Settings for the host.
3. Locate the parameter Numa.LocalityWeightActionAffinity.
4. Set the value to 0 to disable the affinity requirement.
5. Alternatively, for a specific VM, add the following entry to the VMX configuration file: numa.LocalityWeightActionAffinity = "0"
6. Monitor the NMIG counters in ESXTOP to verify that locality migrations have ceased.

Note: In VCF/vSphere 9.1 and later, this setting is disabled by default along with other scheduler enhancements to mitigate migration overhead.

Additional Information

NUMA nodes are heavily load imbalanced, causing performance issues.