VM nvdimm config options for NUMA
Article ID: 337060
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
This article provides best practices for PMem when non-deterministic NVDIMM performance is caused by sub-optimal NUMA locality.
By default, PMem allocation for virtual machines NVDIMMS does not consider NUMA. This can result in the VM running on a different NUMA node than PMEM is allocated from, which causes NVDIMM accesses from the VM to be remote, resulting in sub-optimal performance.
Once PMem has been initially allocated for a VM, it remains on its NUMA node. The virtual machine NUMA clients will be initially placed and migrated according to constrains by the NUMA scheduler, which does not include PMem locality. Baring any contention or other workloads, this might be round-robin on every VM power on.
By default, PMem allocation for virtual machines NVDIMMS does not consider NUMA. This can result in the VM running on a different NUMA node than PMEM is allocated from, which causes NVDIMM accesses from the VM to be remote, resulting in sub-optimal performance.
Once PMem has been initially allocated for a VM, it remains on its NUMA node. The virtual machine NUMA clients will be initially placed and migrated according to constrains by the NUMA scheduler, which does not include PMem locality. Baring any contention or other workloads, this might be round-robin on every VM power on.
Environment
VMware vSphere 7.0.x
VMware vSphere 6.7.x
VMware vSphere 6.7.x
Resolution
To affinitize a virtual machines vCPUs and memory to a physical NUMA node, you can set the VM Advanced Option numa.nodeAffinity = X, where X equals the physical NUMA (starting at 0). Affinitizing the VM to the same node as the PMem allocation will result in optimal performance.
To control the locality of NVDIMM devices, ESXi 6.7 EP14 (15820472) and 7.0b/P01 (16324942) introduced NVDIMM nodeAffinity.
For a host with 2 physical NUMA nodes and a wide VM across both, you might want to set a direct affinity for the vCPUs in vNUMA node 0 to pNUMA node 0, and vice versa for vNUMA node 1. For that, set the VM Advanced Option sched.vcpuX.affinity = Y, where X is the vCPU and Y the PCPU. Note that from 7.0 U2m onwards, ESXi supports per vNUMA node affinity. sched.nodeX.affinity = Z, where X is the vNUMA node and Z the physical NUMA node. To configure the NVDIMM affinity, set nvdimm0:X.nodeAffinity = Z, where X is the device number and Z the PMem node used for allocation (or nodes comma separated if multiple). A VM on ESXi 7.0 U2 or later that is wide over 2 physical NUMA nodes and NVDIMM devices on two PMem NUMA nodes, might have the following configuration for deterministic locality from each NUMA client to the PMem allocation:
To control the locality of NVDIMM devices, ESXi 6.7 EP14 (15820472) and 7.0b/P01 (16324942) introduced NVDIMM nodeAffinity.
For a host with 2 physical NUMA nodes and a wide VM across both, you might want to set a direct affinity for the vCPUs in vNUMA node 0 to pNUMA node 0, and vice versa for vNUMA node 1. For that, set the VM Advanced Option sched.vcpuX.affinity = Y, where X is the vCPU and Y the PCPU. Note that from 7.0 U2m onwards, ESXi supports per vNUMA node affinity. sched.nodeX.affinity = Z, where X is the vNUMA node and Z the physical NUMA node. To configure the NVDIMM affinity, set nvdimm0:X.nodeAffinity = Z, where X is the device number and Z the PMem node used for allocation (or nodes comma separated if multiple). A VM on ESXi 7.0 U2 or later that is wide over 2 physical NUMA nodes and NVDIMM devices on two PMem NUMA nodes, might have the following configuration for deterministic locality from each NUMA client to the PMem allocation:
sched.node0.affinity = 0 sched.node1.affinity = 1 nvdimm0:0.nodeAffinity = 0 nvdimm0:1.nodeAffinity = 1
As with all manual affinities, care in placement and operation is necessary.
While unrelated to locality, a further optimization for fully deterministic performance is to pre-allocate the entire NVDIMM during VM power on. This prevent runtime allocation overhead when the VM uses an NVDIMM page for the first time. While the VM is powered off, set they VM Advanced Option sched.pmem.prealloc = TRUE.
Additional Information
Feedback
Yes
No
Powered by
