Packets loss between T1 and T0 gateway with misconfigured QoS profile

Products

• VMware NSX

Issue/Introduction

• Packets are received on the uplink interface of Tier-1 But not arrive at the downlink interface of Tier-0
• The T1 uplink interface shows drops on Rate-Limit under TX-Drops by command

  get logical-router interface <uuid> stats

  ... ...

  TX-Drops    : 713
        Blocked     : 0
        Firewall    : 0
        Frag-Needed : 0
        No-neighbor : 0
        No-Memory   : 0
        No-Linked-Port: 0
        Rate-Limit  : 713

 

Environment

VMware NSX-T Data Center
VMware NSX-T 4.x

Cause

Gateway QoS profile is applied on the Tier-1 and the Burst Size is misconfigured

Resolution

Remove the Gateway QoS profile OR configure Committed Bandwidth / Burst Size with proper value

Additional Information

Command "get logical-router interface <uuid> stats" unable to see the Rate-Limit on NSX-T instead using "get logical-router interface <uuid> stats |json" to check tx_drop_rate_limit



These KBs which might come handy below for applying the QoS and the burst size for both ingress and egress traffic.

 

Packets loss between T1 and T0 gateway with misconfigured QoS profile
https://knowledge.broadcom.com/external/article/390821/packets-loss-between-t1-and-t0-gateway-w.html

Configuring Gateway QoS
https://knowledge.broadcom.com/external/article/324160/configuring-gateway-qos.html

Note : Gateway QoS profile is supported only on tier-1 gateways.

 

• From your browser, log in with admin privileges to an

  NSX Manager

  at https://<nsx-manager-ip-address>.
• Select

  Networking

  > Networking Settings

  .
• Click the

  Gateway QoS Profiles

  tab.
• Click

  Add Gateway QoS Profiles

  .
• Enter a name for the profile.
• Enter the committed bandwidth limit that you want to set for the traffic.
• Enter the burst size. Use the following guidelines for burst size.

   

  - B

    is the burst size in bytes.

   

  - R 

  is the committed rate (or bandwidth) in Mbps.

   

  - I  

  is the time interval in milliseconds, to refill or withdraw tokens(in bytes) from the token bucket. Use the

  get dataplane

  command from the

  NSX Edge

  CLI to retrieve the time interval,

  
  Qos_wakeup_interval_ms

  . The default value for

  Qos_wakeup_interval_ms

  is 50ms. However, this value is automatically adjusted by the dataplane based on the QoS configuration.

   

  The constraints for burst size are:
  

   
• B >= R * 1000,000 * I / 1000 / 8

  because burst size is the maximum amount of tokens that can be refilled in each interval.

   
• B >= R * 1000,000 * 1 / 1000 / 8

  because the minimum value for

  I

  is 1 ms, taking into account dataplane CPU usage among other constraints.

   
• B >= MTU of SR port

  because at least the MTU-size amount of tokens need to be present in the token bucket for an MTU-size packet to pass rate-limiting check.

   

  Since the burst size needs to satisfy all three constraints, the configured value of burst size would be
  Max (R * 1000,000 * I / 1000 / 8, R * 1000,000 * 1 / 1000 / 8, MTU)

   

  For example, if

    R

  = 100 Mbps,

  I

  = 50 ms, and

  MTU

  = 1500, then

  B >= max (100 * 1000,000 * 50 / 1000/ 8, 100 * 1000,000 * 50 / 1000/ 8, 1500) = 625000 in bytes
  
  Click

  Save

  .