NUMA imbalance when running a single NSX Edge VM on an ESX host

Products

• VMware NSX

Issue/Introduction

The ESX CPU scheduler typically optimizes for cache locality by keeping a VM and its associated network kernel threads on the same NUMA node. However, in scenarios where a host runs a single VM handling high traffic, this can lead to increased CPU contention on that NUMA node, resulting in packet drops at both VNICs and PNICs.

 

You can identify CPU contention by observing high core utilization and increased CPU ready times using the following methods:

1. Check ESXi Core Utilization:
   
   Run esxtop and examine the ""CORE UTIL"" section. The ""NUMA"" metric shows the CPU load of each NUMA node.
   
   
   
   
2. Next check CPU Ready Time: 
   
   Run the following command on the ESXi host CLI :
   

   Run net-stats -i 30 -t icWQv -A.

   Examine the output, the ""Ready time"" metric under vcpus section, indicates the percentage of time a VCPU or kernel thread waits for CPU access.
   
   For example:
   "vcpus": {
   ""2097948"": {""id"": 2097948, ""used"": 0.00, ""ready"": 5.00, ""cstp"": 0.00, ""name"": ""vmnic4-pollWorld-39-0x#####8f340"", ""sys"": 0.00, ""sysoverlap"": 0.00, ""limited"": 0.00, ""vmkcall"": 0.00, ""actnot"": 0.00, ""migtot"": 0, ""miginterl3"": 0}
   
   A ready time greater than 3-4% suggests possible CPU contention.
   
   
3. Packet drops can also be identified using the same net-stats command:
   
   Packet Drops at PNIC: The rxeps metric indicates packet drops averaged over the collection period.
   
   {""name"": ""vmnic1"", ""switch"": ""DvsPortset-0"", ""id"": 2214592528, ""mac"": ""6c:##:##:4d:da:b1"", ""rxmode"": 0, ""tunemode"": 2, ""uplink"": ""true"", ""ens"": ""true"", ""promisc"": ""false"", ""sink"": ""false"" , ""txpps"": 4077789, ""txmbps"": 10700.1, ""txsize"": 327, ""txeps"": 0.00, ""rxpps"": 9248959, ""rxmbps"": 19977.8, ""rxsize"": 269, ""rxeps"": 0.00, ""portstats"": {""txpps"": 4077827, ""txeps"": 0.00, ""rxpps"": 9249037, ""rxeps"": 0.02 }, ""upt"": ""false"", ""modeTransitions"": 0, ""intr"": {""count"": 365797, ""used"": 26.58},
   ""vmnic"": {""devname"": ""vmnic1.bnxtnet"", ""driver version"": ""232.0.18.0"", ""firmware version"": ""233.0.196.0 /pkg 23.31.18.10"", ""device NUMA node"": 0, ""speed"": 25000, ""txpps"": 3327436, ""txmbps"": 8837.7, ""txsize"": 331, ""txeps"": 0.00, ""rxpps"": 9248840, ""rxmbps"": 20273.5, ""rxsize"": 273, ""rxeps"": 57866.32 }}
   
   In this example we can see that the rxeps value is high ---> rxeps: 57866.32 → This indicates a high rate of packet drops at the physical NIC level. 
   
   
   
4. Packet Drops at VNIC: The rxeps metric indicates packet drops averaged over the collection period.

{""name"": ""NewEdge1.eth2"", ""switch"": ""DvsPortset-0"", ""id"": 67108890, ""mac"": ""00:##:##:bf:7b:90"", ""rxmode"": 0, ""tunemode"": 0, ""uplink"": ""false"", ""ens"": ""true"", ""promisc"": ""false"", ""sink"": ""false"" ,
  ""txpps"": 4077816, ""txmbps"": 10700.2, ""txsize"": 327, ""txeps"": 0.00, ""rxpps"": 8727633, ""rxmbps"": 18851.7, ""rxsize"": 269, ""rxeps"": 84248.27,"

In this example we can see that the rxeps value is high ---> rxeps: 84248.27 → This indicates a high rate of packet drops at the virtual NIC level. 

Environment

VMware NSX

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.