Network latency observed on single NSX-V Edge node with high amount of network traffic

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Latency is seen with network traffic passing through an NSX-V Edge node 
• High amounts of network traffic passing through an Edge Node
• DVS Network I/O Control(NIOC) is enabled 

Environment

VMware vSphere ESXi 7.x

Cause

NIOC is a shared network control feature, if you have a single VM on a DVS pushing a high amount of traffic the shares may not be enough to cover the required packet rate. 

Resolution

If you do not have more than 1 VM utilizing high bandwidth disable NIOC for the shares to be fully maximized on the DVS 

 

Or 

Increase the global queue setting on the host. The advanced option that you want to change is NetSchedHClkLeafQueueDepthPktMAX.  The default value is 5120 and the recommended setting is to increase it to 10000 on all hosts that might run an NSX edge. 
 

vsish -e set /config/Net/intOpts/NetSchedHClkLeafQueueDepthPktMAX 10000


vsish -e get /config/Net/intOpts/NetSchedHClkLeafQueueDepthPktMAX
Vmkernel Config Option {
   Default value:5120
   Min value:1
   Max value:20480
   Current value:5120
   hidden config option:1
   Description:Maximum number of packets each HClk leaf node can hold
 

vsish -e get /config/Net/intOpts/NetSchedHClkLeafQueueDepthPktMAX
Vmkernel Config Option {
   Default value:5120
   Min value:1
   Max value:20480
   Current value:10000
   hidden config option:1
   Description:Maximum number of packets each HClk leaf node can hold
   Host specific config option:0
   Option update requires reboot:0
   Option update requires maintenance mode:0

 

Additional Information

Impact/Risks:

Latency is seen in the network when passing through an NSX-V Edge