Advanced configuration settings to create or update NSX Edge Transport node to fine tune the performance.

Products

VMware

Issue/Introduction

This article provides details on fine tuning the advanced configurations for performance parameters on Edge Transport nodes.

Resolution

REST api

METHOD	URI
POST	https://<NSXMgrIP>/api/v1/transport-nodes/
POST	https://<NSXMgrIP>/api/v1/transport-nodes/<edge-id>?action=redeploy

Payload:

{
"node_settings": {
"advanced_configuration": [

]
}

Coalescing Scheme and Param (Supported from 4.1.0)

Coalescing is enabled by default on all the nics of the edge transport node. The default scheme is rbc and the value is set to 4000. This adds to the latency and if the customer is running latency sensitive workload, using this we can either disable this feature which will reduce the latency, or change the value of the interrupt to some higher values. Depending on a particular setup or perf bottleneck, we might use coalescing where we see too frequent or too less interrupts being posted.

Payload:

{
"node_settings": {
"advanced_configuration": [
    {
         "key": "advanced-config:coalescingScheme",
         "value": "rbc"

    },{
         "key": "advanced-config:coalescingParams",
         "value": "4000"

    }
]
}

RX and TX ring size

Network Interface Cards (NICs) use rings to receive and send packets. The rings contain pointers to the memory locations that the NIC uses to receive (rx) or transmit (tx) the packets. Larger ring sizes may provide a buffer to avoid packet loss when the CPU or the link is busy. The minimum ring size is 256, and the maximum is 4096. Small ring sizes are not recommended with fast interfaces.

Payload:

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "ovf-param:nsx_edge_tx_ring_size",
           "value": "4096"
    },{
           "key": "ovf-param:nsx_edge_rx_ring_size",
           "value": "4096"
    }
]
}

Latency sensitivity settings:

At edge virtual machine level, user can set the latency sensitivity of the edge virtual machine which defaults to Normal. Setting this to High can yield significantly lower latencies and jitter, as a result of the following mechanisms that take effect in ESXi :-

Exclusive access to physical resources, including pCPUs dedicated to vCPUs with no contending threads for executing on these pCPUs.
Full memory reservation eliminates ballooning or hypervisor swapping leading to more predictable performance with no latency overheads due to such mechanisms.
Halting in the VM Monitor when the vCPU is idle, leading to faster vCPU wake-up from halt, and bypassing the VMkernel scheduler for yielding the pCPU. This also conserves power as halting makes the pCPU enter a low power mode, compared to spinning in the VM Monitor with the monitor_control.halt_desched=FALSE option.
Disabling interrupt coalescing and LRO automatically for VMXNET 3 virtual NICs.
Optimized interrupt delivery path for VM DirectPath I/O and SR-IOV passthrough devices, using heuristics to derive hints from the guest OS about optimal placement of physical interrupt vectors on physical CPUs.

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "advanced-config:sched.cpu.latencySensitivity",
           "value": "High"
    }
]
}

Enable dynamic core allocation.

Enable dynamic core allocation is applicable to only edges with LARGE form factor. Dynamic core allocation is required from Edge platform to support CPU-heavy security features such as TLS Inspection and IDPS on GFW. If Dynamic core allocation is disabled, then TLS Inspection /IDPS performance would significantly degrade. Currently, dynamic core allocation cannot coexist with VPN ESP RSS. The acceptable values are either true or false.

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "ovf-param:nsx_edge_dynamic_core_enabled",
           "value": "true"
    }
]
}

Datapath Enhanced Service Core Mode Enabled

Datapath Enhanced Service Core Mode Enabled indicates whether the "Enhanced Service Core mode" is enabled for Edge datapath. When this option is enabled, data plane service will allocate additional cores for services running on the edge transport node. If this option is disabled, then data plane service half the number of cores allocated for fast path to the services running on the edge appliance. Applicable to edges with XLarge form factor only, ignored on other sizes. The acceptable values are either true or false.

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "ovf-param:nsx_edge_enhanced_service_core_mode_enabled",
           "value": "true"
    }
]
}

Affinity Rule

User can create anti affinity rules at vsphere cluster and mention those rule name here in advanced configuration and the edge virtual machine will be deployed abiding this rule. Acceptable values display name of the anti affinity rule created at the vsphere cluster on which the edge is getting deployed. If the referenced affinity rule is not found at VCenter server, Edge virtual machine deployment workflow will fail. Edge virtual machine would not proceed with further Post deployment tasks and is stuck in powered off state.

{
"node_settings": {
"advanced_configuration": [
    {
          "key": "affinity-target:rule",
           "value": "nsx-edge-anti-affinity-rule"
    }
]
}