NUMA nodes are heavily load imbalanced causing high contention for some virtual machines

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Some NUMA nodes/PCPUs are underutilized while others are overloaded.
• Some of the virtual machines on over-utilized NUMA node(s) may display a higher ready time than virtual machines on the lower utilized node(s).
• Some of the virtual machines are migrating often between the over and under utilized NUMA node(s).

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 5.0
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1

Cause

This issue occurs when the CPU scheduler considers the relationship between scheduling contexts and places them closely, NUMA clients might be placed on the same NUMA node due to frequent communication between them. Such relation or action-affinity is not limited between NUMA clients and can also be established with I/O context. This policy may result in a situation where multiple virtual machines are placed on the same NUMA node, while there are very few on other NUMA nodes. While it might seem to be unbalanced and an indication of a problem, it is expected behavior and generally improves performance even if such a concentrated placement causes non-negligible ready time. Note that the algorithm is tuned in a way that high enough CPU contention will break such locality and trigger migration away to a less loaded NUMA node. If the default behavior of this policy is not desirable, this action-affinity-based migration can be turned off by overriding the given advanced host attribute.

Resolution

If your workload does not benefit from the additional locality and the increased contention is negatively affecting performance, you can turn off Action Affinity by setting the /Numa/LocalityWeightActionAffinity as 0.

To set /Numa/LocalityWeightActionAffinity as 0:

1. Browse to the host in the vSphere Web Client navigator.
2. Click the Manage tab.
3. Click Settings in the Manage tab.
4. In System, click Advanced System Settings.
5. In Advanced System Settings, select the Numa.LocalityWeightActionAffinity.
6. Click the Edit button to edit the value.
7. Type 0 as the value.
8. Click OK.