Advanced configuration settings to create or update NSX Edge Transport node to fine tune performance.

Products

VMware NSX

Issue/Introduction

This article provides details on fine tuning the advanced configurations for performance parameters on Edge Transport nodes.

Environment

VMware NSX

Resolution

REST api

METHOD	URI
POST	https://<NSXMgrIP>/api/v1/transport-nodes/
POST	https://<NSXMgrIP>/api/v1/transport-nodes/<edge-id>?action=redeploy

Payload:

{
"node_settings": {
"advanced_configuration": [

]
}

Coalescing Scheme and Param (Supported from 4.1.0)

Coalescing is enabled by default on all the nics of the edge transport node. The default scheme is rbc and the value is set to 4000. This adds to the latency and if the customer is running latency sensitive workload, using this method it can either disable this feature which will reduce the latency, or change the value of the interrupt to some higher values. Depending on a particular setup or perf bottleneck, coalescing may be used where there are too frequent or too less interrupts being posted.

Payload:

{
"node_settings": {
"advanced_configuration": [
    {
         "key": "advanced-config:coalescingScheme",
         "value": "rbc"

    },{
         "key": "advanced-config:coalescingParams",
         "value": "4000"

    }
]
}

RX and TX ring size

Network Interface Cards (NICs) use rings to receive and send packets. The rings contain pointers to the memory locations that the NIC uses to receive (rx) or transmit (tx) the packets. Larger ring sizes may provide a buffer to avoid packet loss when the CPU or the link is busy. The minimum ring size is 256, and the maximum is 4096. The value must be a power of 2 - 256, 512, 1024, 2048, or 4096.

Note: Small ring sizes are not recommended with fast interfaces.

Payload:

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "ovf-param:nsx_edge_tx_ring_size",
           "value": "4096"
    },{
           "key": "ovf-param:nsx_edge_rx_ring_size",
           "value": "4096"
    }
]
}

Latency sensitivity settings:

At edge virtual machine level, user can set the latency sensitivity of the edge virtual machine which defaults to Normal. Setting this to High can yield significantly lower latencies and jitter, as a result of the following mechanisms that take effect in ESXi :-

Exclusive access to physical resources, including pCPUs dedicated to vCPUs with no contending threads for executing on these pCPUs.
Full memory reservation eliminates ballooning or hypervisor swapping leading to more predictable performance with no latency overheads due to such mechanisms.
Halting in the VM Monitor when the vCPU is idle, leading to faster vCPU wake-up from halt, and bypassing the VMkernel scheduler for yielding the pCPU. This also conserves power as halting makes the pCPU enter a low power mode, compared to spinning in the VM Monitor with the monitor_control.halt_desched=FALSE option.
Disabling interrupt coalescing and LRO automatically for VMXNET 3 virtual NICs.
Optimized interrupt delivery path for VM DirectPath I/O and SR-IOV passthrough devices, using heuristics to derive hints from the guest OS about optimal placement of physical interrupt vectors on physical CPUs.

IMPORTANT: The Edge VM must be configured with 100% CPU and Memory reservations before attempting to apply the following configuration.

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "advanced-config:sched.cpu.latencySensitivity",
           "value": "High"
    }
]
}

Datapath Enhanced Service Core Mode Enabled

Datapath Enhanced Service Core Mode Enabled indicates whether the "Enhanced Service Core mode" is enabled for Edge datapath. When this option is enabled, data plane service will allocate additional cores for services running on the edge transport node. If this option is disabled, then data plane service half the number of cores allocated for fast path to the services running on the edge appliance. Applicable to edges with XLarge form factor only, ignored on other sizes. The acceptable values are either true or false.

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "ovf-param:nsx_edge_enhanced_service_core_mode_enabled",
           "value": "true"
    }
]
}

Affinity Rule

User can create anti affinity rules at vsphere cluster and mention those rule name here in advanced configuration and the edge virtual machine will be deployed abiding this rule. Acceptable values display name of the anti affinity rule created at the vsphere cluster on which the edge is getting deployed. If the referenced affinity rule is not found at VCenter server, Edge virtual machine deployment workflow will fail. Edge virtual machine would not proceed with further Post deployment tasks and is stuck in powered off state.

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "affinity-target:rule",
           "value": "nsx-edge-anti-affinity-rule"
    }
]
}