Understanding Packet Drops on VLAN Segment Logical Switch ports inside an Edge

Products

VMware NSX

Issue/Introduction

- Segment Statistics (visible after expanding Segment in NSX UI > select View Statistics) for the VLAN Segment that T0 uplinks are connected to may show a high number of dropped Tx packets
- vRNI reports high Tx drops on that VLAN Logical Switch and its Logical Switch Ports

- Logical Switch port statistics in the Edge CLI show 0 drops:

nsx-edge> get logical-switches | find <Logical Switch / Segment name>
nsx-edge> get logical-switch <UUID of Logical Switch that T0 uplinks are connected to> ports stats
Logical Switch
UUID        : <UUID>
VLAN        : <VLAN>
Device      : fp-eth<#>
IFUID       : <#>
Enable Hub : False
Ports
    Port        : <UUID>
    RX-Packets : 30353136097
    RX-Bytes    : 29598601841310
    RX-Drops    : 0
        Malformed   : 0
        No-Match    : 0
        L2-Loop     : 0
    TX-Packets : 121228576439
    TX-Bytes    : 19316183547993
    TX-Drops    : 0                              <-------------------------- 0 Tx drops shown on the LS port
        No-Memory   : 0
        No-Linked-Port: 0

- Querying the Management plane API shows a high number / percentage of drops for the same Logical Switch port UUID:

root@nsxmgr:~# curl -v -k -u admin -H "Content-Type:application/json" -X GET https://<NSX mgr IP>/api/v1/logical-ports/<port UUID>/statistics
{
"logical_port_id" : "<UUID>",
"last_update_timestamp" : <UNIX timestamp>,
"rx_packets" : {
    "dropped" : 335566,
    "total" : 30513571612
},
"rx_bytes" : {
    "total" : 29605186972550
},
"tx_packets" : {
    "dropped" : 907865596,        <------------------------- High Tx drop count on LS port when polling Management Plane API
    "total" : 21446967719
},
"tx_bytes" : {
    "total" : 1966247116597

Cause

Logical Switch Ports connected to T0 uplink Logical Router interfaces are transmitting packets south, to their peer Logical Router port.

Logical Switch port and Logical Router port statistics in the Edge CLI report statistics for their respective objects.
VLAN Logical Switch ports inside an Edge will not drops any packets themselves unless there is no connected port.

However, the Management plane API output for Logical Switch ports combines the drop statistics from the Logical Switch port and its peer (Logical Router) port. This was done since the Logical Switch port itself does not drop packets, and the statistics would not show any useful information.

The Management Plane API is polled to populated Segment Statistics in the NSX UI, as well as graphs and statistics in VRNI.

Resolution

If Logical Switch port Tx drops are high, review the Rx drops on the peer Logical Router port.

nsx-edge> get logical-switch <UUID of Logical Switch that T0 uplinks are connected to> ports
Logical Switch
UUID        : <LS UUID>
VLAN        : <#>
Device      : fp-eth <#>
IFUID       : <#>
Enable Hub : False
Ports
    Port        : <LS port UUID>
    IFUID       : <#>
    Peer        : <LR port UUID>              <----------------------- peer Logical Router port

Review Logical Router port stats:

nsx> get logical-router interface <LR port UUID> stats
interface   : <LR port UUID>
ifuid       : <#>
VRF         : <VRF UUID>
name        : <Interface name>
IP/Mask     : <Interface IP/mask>
MAC         : <Interface MAC>
VLAN        : <#>
LS port     : <LS port UUID>
urpf-mode   : NONE
admin       : up
op_state    : up
MTU         : 1500
statistics
    RX-Packets : 21274601874
    RX-Bytes    : 19365401923743
    RX-Drops    : 907739502                       <------------------------ LR port Rx drops, which explain the LS port Tx drops
        Blocked     : 0
        DST-Unsupported: 0
        Firewall    : 8812
        Malformed   : 0
        No-Receiver : 21
        No-Route    : 0
        RPF-Check   : 0
        Protocol-Unsupported: 240254
        IPv6        : 2
        Port-Unsupported: 0
        TTL-Exceeded: 236
        Kni         : 907490177
        IPsec       : 0
        IPsec-NoSA : 0
        IPsec-NoVTI : 0
    TX-Packets : 30448928408
    TX-Bytes    : 29712039841325
    TX-Drops    : 311001
        Blocked     : 0
        Firewall    : 311001
        Frag-Needed : 0
        No-neighbor : 0
        No-Memory   : 0
        No-Linked-Port: 0
        IPsec       : 0
        IPsec-NoSA : 0
        IPsec-NoVTI : 0
        IPsec-Policy-Error: 0
        IPsec-Policy-Block: 0
    IP Ressemble
        Fragments-OK: 1681
        Fragemnts-Error: 0
        Fragments-Timeout: 10
    IP Fragment
        Fragments-OK: 38073606
        Fragments-Error: 0

In the above example, the Kni Rx drop count is high. In the case of unicast drops, the Kni category indicates that there was more traffic than the linux kernel can handle. In the case of multicast drops, it indicates that packets were not sent to the kernel.