Advanced configuration settings to create or update NSX Edge Transport node to fine tune performance.

Products

• VMware NSX

Issue/Introduction

• This article provides details on advanced fine tuning parameters that are now supported for the Edge VM, which can be applied to improve the performance parameters on Edge Transport nodes.
• This article should not be viewed as a best practice guide for NSX Edge VM performance tuning. The parameters could be considered when trying to improve performance for latency sensitive workloads. It is recommended that they are tested in a test environment, similar to the production environment, prior to applying them in the production environment.
• Please note that these parameters are not candidates for general-purpose usage on the Edge VM. Also, each advanced parameter should be considered individually or in combination depending on what needs to be achieved.  As of now it is recommended that customers should refer to the 'ESX and vSphere Best practices guide', and the 'Configure Advanced NSX Edge Parameters' section of the NSX Administration guide.
  
  

Environment

VMware NSX

Resolution

• Please note in most cases the advanced configurations discussed in this KB are unavailable via UI and must be applied via API.
• For planned new Edge deployments, the Edge can be deployed via API

• • POST https://<NSXMgrIP>/api/v1/transport-nodes/
• For existing Edges, they will need to be redeployed via API to update their settings. Note new Edges can alternatively be deployed from UI first and then redeployed via API to apply the setting.

• • POST https://<NSXMgrIP>/api/v1/transport-nodes/<edge-id>?action=redeploy
• In all cases the API body must include the advanced configuration section framework.

• More detailed information on these API's can be found in the 'Configure Advanced NSX Edge Parameters' section of the NSX Administration guide .

{
"node_settings": {
"advanced_configuration": [
    
]
}

Coalescing Scheme and Param (Supported from 4.1.0)

• Interrupt coalescing is enabled by default on all the vnics of the Edge VM Transport Node. The default scheme is rbc and the value is set to 4000.
• Virtual network interrupt coalescing can reduce the number of CPU interrupts, thus potentially decreasing CPU utilization.
• However, the feature can add to network latency, so if running latency sensitive workloads is the priority, then it is worth considering disabling the feature, changing the value of the interrupt to some higher values or one of the other options available. 
• Please read the 'Configure Advanced NSX Edge Parameters' section of the NSX Administration guide for more details on how to configure this feature, and the available options.
• Important not to try manually configure Interrupt Coalescing if configuring VM latency sensitivity to high. More information on Interrupt Coalescing can be found here. 
  
  

  Payload:

  { "node_settings": { "advanced_configuration": [     {          "key": "advanced-config:coalescingScheme",          "value": "rbc"       },{          "key": "advanced-config:coalescingParams",          "value": "4000"       } ] }

 

RX and TX ring size

• Network Interface Cards (NICs) use rings to receive and send packets.
• The rings contain pointers to the memory locations that the NIC uses to receive (rx) or transmit (tx) the packets.
• Larger ring sizes may provide a buffer to avoid packet loss when the CPU or the link is busy.
• The minimum ring size is 256, and the maximum is 4096.  The value must be a power of 2 - 256, 512, 1024, 2048, or 4096.
• Ring size can be configured using the API and the following payload:

Payload:

{ "node_settings": { "advanced_configuration": [     {           "key": "ovf-param:nsx_edge_tx_ring_size",            "value": "4096"     },{            "key": "ovf-param:nsx_edge_rx_ring_size",            "value": "4096"     } ] }

Note: Small ring sizes are not recommended with fast interfaces.

• Alternatively, it can be configured using the Edge command line interface. Please read this KB for detail on how that can be done.

 

Latency sensitivity settings:

• At edge virtual machine level, latency sensitivity can be set to high on the edge virtual machine which defaults to Normal. Setting this to High can yield significantly lower latencies and jitter, as a result of the following mechanisms that take effect in ESXi :-

• • Causes the CPU scheduler to attempt to exclusively assign one physical CPU core for each vCPU configured on the Edge VM. Exclusive access results in no contending threads on the pCPU's the Edge VM's are running on. Please note that the assigned physical cores are no longer available to other virtual machines or vmkernel worlds, while those pCPU's are being used by the Edge.

  • Full memory and CPU reservations must be configured on the Edge VM before attempting to change the setting to high. Setting a memory reservation eliminates ballooning or hypervisor swapping leading to more predictable performance with no latency overheads due to such mechanisms.

  • Disabling interrupt coalescing and LRO automatically for VMXNET 3 virtual NICs. Important not to try manually Interrupt Coalescing if configuring VM latency sensitivity to high. 

• Please read section 'Running Network Latency Sensitive Workloads' in the vSphere 8.0 Best Practice Performance Guide for more information.
• If ample free physical CPU capacity is available on your hosts, then it is recommended to set this to high if latency improvements are the priority.
• The setting can be configured using the deploy\redploy API and the payload specified below or in vCenter using the steps in the KB. Important not to try manually configure Interrupt Coalescing if configuring VM latency sensitivity to high. 

Payload:

{ "node_settings": { "advanced_configuration": [     {           "key": "advanced-config:sched.cpu.latencySensitivity",            "value": "High"     } ] }

 

Datapath Enhanced Service Core Mode Enabled

• Datapath Enhanced Service Core Mode can be enabled for the Edge data path.
• Configurable on XLarge form factor only, ignored on other size edge form factors.
• By default an XL is configured with 16 * vCPU's, 8 * vCPU's are dedicated to fast path, 4 * vCPU's are for services, and 4 * general Linux vCPU's. The service cores have pinned datapath threads for services like L7 FW, but are not isolated like the fast path cores, and thus are subject to normal Linux scheduling.
• When this option is enabled, data plane service will allocate additional cores for services running on the edge transport node. An example of such services would be Layer 7 Firewall. 
• In enhanced service core mode, there are 4 * fast path vCPU's, 8 * service vCPU's, and 4 * general Linux vCPU's. Four fast path cores is usually more than adequate if the packet rate is relatively low, and the bottleneck is in the services, so it would only be used in specific use cases.
• It can be configured using the following payload snip with the API. The acceptable values are either true or false (default). 

{ "node_settings": { "advanced_configuration": [     {           "key": "ovf-param:nsx_edge_enhanced_service_core_mode_enabled",            "value": "true"     } ] }

 

Affinity Rule

• There may be further performance improvements by ensuring two Edge VM's are not running on the same host at the same time.
• Two Edge VM's on the same host with a high PPS rate may end up in a state of contention. In this case the contention could be the limited number of vmkernel RX threads available.
• Anti-affinity rules created at vsphere cluster could be used to enforce this behaviour.
• The rule name can be configured in advanced settings using the deploy\redeploy API and the payload below.
• An acceptable values is the display name of the anti affinity rule created at the vsphere cluster on which the edge is being deployed.
• If the referenced affinity rule is not found at VCenter server, then the Edge virtual machine deployment workflow will fail. Edge virtual machine would not proceed with further Post deployment tasks and is stuck in powered off state.

{ "node_settings": { "advanced_configuration": [     {           "key": "affinity-target:rule",            "value": "nsx-edge-anti-affinity-rule"     } ] }

 

RSS (Receive Side Scaling)

• RSS is a TCP enhancement, that enables the use of multiple cores on the receive side to process incoming traffic.
• Without RSS, ESX by default will use only one core to process incoming traffic. Utilizing only one core has a huge impact on the overall throughput as the
  receiving node then becomes the bottleneck.
• RSS on the NIC creates multiple queues to process incoming traffic and efficiently uses a core for each queue, with most NIC cards being able to support at least 4 queues.
• RSS can be configured on the pNIC level and on the level of the Edge vNIC's. 
• How it is configured on the pNIC level is specific to the NIC manufacturer, but is usually done through the ESXi command line.
• To configure it on the Edge VM level, the VM's vmx file to have the following parameter included: ethernet3.pnicFeatures = "4" .
• For more details on how to configure RSS and for additional information on how to tune an Edge, then I would recommend reading the latest version of the 'NSX Reference Design Guide'.