Higher network latency on virtual machines on vSphere ESXi 7.0.x configured with passthrough NICs and more than 8 TB Memory

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Virtual machines running on vSphere ESXi 7.0.x configured with memory more than 8 TB (pre-allocated only) and passthrough NICs may encounter higher network latency due to misaligned NUMA node memory segments.

This does not affect virtual machines configured with VMXNET3 NICs or with memory allocation turned off.

Environment

VMware vSphere 7.0.x

Cause

This issue occurs due to incorrect NUMA memory calculation, the NUMA memory ends up the virtual machine in a misaligned state.

Resolution

The issue will be fixed in future vSphere releases. Meanwhile, you can use the attached Hot Patch to this article "VMware-ESXi-7.0hp-19457183-depot.zip"

For more information about installing the Hot Patch , see “esxcli software vib” commands to patch an ESXi 6.x/7.x host (2008939)

Workaround:

To workaround this issue it is required to modify the VM memory configuration that the memory allocation code is satisfied and doesn't modify the NUMA memory map during pre-allocation.

To define and set (see picture edit VM setting) the correct VM memory configuration following procedure can get used:

The process is to divide the configured VM memory (see below picture) of the SAP HANA VM by the vSphere used memory segment size (32768). If the result is evenly divisible by 32768 and less than 512 then the algorithm won’t change the NUMA memory map and cause the experienced issue.

Example calculation:

• Assumption, initial VM memory was set to 12222464 MB
  VM memSize = "12222464 " 
  numa.memory.gransize = "32768"
• Calculation:
  12222464 / 32768 = 373 / 8 socket = 46,625, which will be an issue since the algorithm now would try to create a wrong NUMA memory map.
  Rounding down the result of 46,625 to 46 and multiplying it with the 8-socket (in our case) and then with 32768 results in the new, aligned VM memory figure of 12058624 MB for this specific VM (46 x 8 = 368 x 32768 = 12058624).

 

In addition to this, the VM advanced configuration parameter numa.memory.gransize = "32768" hast to get set in the “Configuration Parameters” section of the VM, see picture below:
 

Verification of the aligned VM memory size for PT enabled VMs:

To verify the correct memory configuration a customer can use the Intel® Memory Latency Checker* to and check the NUMA latency map. Any misaligned memory configurations will get detected by this check and would require changing the VM memory setting.

1st picture shows an example of a NUMA latency map with a misaligned VM memory value. The 2nd picture shows the NUMA map with an aligned memory setting.