Interpreting NSX Edge Interface stats
search cancel

Interpreting NSX Edge Interface stats

book

Article ID: 324167

calendar_today

Updated On:

Products

VMware NSX VMware vDefend Firewall

Issue/Introduction

The NSX Edge cli provides detailed stats for Gateway interfaces. 
This KB provides details on how to interpret these stats and recommended actions that can be taken.

Example of Edge cli
 

Edge1> get logical-router interface 8638183a-####-####-####-##########4e stats
Wed Jan 24 2024 UTC 07:09:38.729
interface   : 8638183a-####-####-####-##########4e
ifuid       : ###
VRF         : 5dfb911e-####-####-####-##########db
name        : tier0-interface-##-##-##-##
IP/Mask     : 10.1.1.2/24;2010::###:####:####:cb7c/64(A);2010::1/64(A);fe80::###:####:####:cb7c/64(A)
MAC         : 00:50:##:##:##:7c
VLAN        : 8
LS port     : 7ea2eb90-####-####-####-##########9f
urpf-mode   : STRICT_MODE
admin       : up
op_state    : up
MTU         : 1500
statistics
    RX-Packets  : 275749
    RX-Bytes    : 23938078
    RX-Drops    : 4996
        Blocked     : 0
        DST-Unsupported: 4515
        Firewall    : 0
        Malformed   : 0
        No-Receiver : 0
        No-Route    : 0
        RPF-Check   : 81
        Protocol-Unsupported: 396
        IPv6        : 4
        Port-Unsupported: 0
        TTL-Exceeded: 0
        Kni         : 0
        Rate-Limit  : 0
        IPsec       : 0
        IPsec-NoSA  : 0
        IPsec-NoVTI : 0
    TX-Packets  : 203743
    TX-Bytes    : 17505975
    TX-Drops    : 0
        Blocked     : 0
        Firewall    : 0
        Frag-Needed : 0
        No-neighbor : 0
        No-Memory   : 0
        No-Linked-Port: 0
        Rate-Limit  : 0
        IPsec       : 0
        IPsec-NoSA  : 0
        IPsec-NoVTI : 0
        IPsec-Policy-Error: 0
        IPsec-Policy-Block: 0
    IP Ressemble
        Fragments-OK: 0
        Fragemnts-Error: 0
        Fragments-Timeout: 0
    IP Fragment
        Fragments-OK: 0
        Fragments-Error: 0



Environment

VMware NSX 4.1.0

Resolution


RX Interface Stats
 

Stats (Rx) Description Action
Blocked Packet dropped due to interface being blocked/admin-down due to SR of the edge node is in standby state

Some packets dropped due to blocked state is expected during failover/fallback case. 

if this counter continue to increase, the following command should be used to check admin and internal_operation:

nsx> get logical-router interface c61a8048-####-####-####-##########44 | json
{
  "admin": "up",
  "internal_operation": "up",
}
DST-Unsupported Packet dropped due to unsupported destination like mcast (but mcast is not enabled), loopback or reserved address

Start capture command can be used to find out the unsupported destination address and where is the traffic coming from (source address)

start capture interface c61a8048-####-####-####-##########44
Firewall Packet dropped due to firewall Check if there is drop rule or state mismatch traffic
Edge1> get firewall b7d57745-####-####-####-##########2b ruleset type firewall rules
Mon Feb 12 2024 UTC 03:43:32.837
Rule count: 1
    Rule ID   : 1005
    Rule      : inout protocol any stateless from any to any accept
 
Edge1> get firewall b7d57745-####-####-####-##########2b ruleset type firewall stats
Mon Feb 12 2024 UTC 03:43:20.547
Rule count: 1
    Rule ID             : 1005
    Input bytes         : 253299828
    Output bytes        : 275434116
    Input packets       : 4189054
    Output packets      : 4607874
    Evaluations         : 8796932
    Hits                : 8796928
    Active connections  : 0
Malformed Packet dropped due to malformed fields, ip checksum or L4 checksum could be the reason of malformed traffic Check the sender checksum offload configuration
No-Receiver Packet dropped due to destination being a lrouter with no receiver i.e. proto not supported by it or no linked tunnel exists Check configuration of lrouter port, if its GRE port, then it should be linked to GRE tunnel port.
root@Edge1:~# edge-appctl -t /var/run/vmware/edge/dpd.ctl lrouter_port/show 2f406d88-####-####-####-#########5d | json_pp
{
   "ifuuid" : "2f406d88-####-####-####-#########5d",
    ...
   "lrouter" : "feca5512-####-####-####-#########89",
    ...
   "name" : "test-#### - 0",
    ...
   "peer" : "37c4f125-####-####-####-#########8c",     >>>> This should be GRE tunnel port UUID
    ...
   "ptype" : "gre-port",
}
 
Edge1> get tunnel-port 37c4f125-####-####-####-#########8c
Mon Feb 12 2024 UTC 03:29:41.339
Tunnel      : 37c4f125-####-####-####-#########8c
IFUID       : 394
LOCAL       : 10.40.40.1
REMOTE      : 10.5.5.5
ENCAP       : GRE
GRETAP VRFID: 2
No-Route Packet dropped due to routing failure or invalid egress port

Check L3 forwarding table for dst IP

nsx> get logical-router 1eef3979-####-####-####-#########6c forwarding
Fri Jan 26 2024 PST 13:55:51.367
Logical Router
UUID                                   VRF    LR-ID  Name                              Type                      
1eef3979-####-####-####-#########6c   2      8      SR-T0-####-####                   SERVICE_ROUTER_TIER0      
IPv4 Forwarding Table
IP Prefix          Gateway IP                                Type        UUID                                   Gateway MAC     
0.0.0.0/0          10.10.138.14                              route       3c71fae6-####-####-####-#########23f   00:1c:##:##:##:95
                   10.10.138.10                                          59b73791-####-####-####-#########cc   00:1c:##:##:##:95
RPF-Check Packet dropped due to no reverse path to destination

Check RPF configuration (urpf-mode) and L3 forwarding table for SRC IP
 

Edge1> get logical-router interface 2f406d88-####-####-####-#########5d
Mon Feb 12 2024 UTC 03:37:45.617
interface   : 2f406d88-####-####-####-#########95d
ifuid       : 395
VRF         : feca5512-####-####-####-##########89
name        : test-#### - 0
mode        : lif
IP/Mask     : 172.16.10.1/24
Fwd-mode    : IPV4_AND_IPV6
MAC         : 02:50:##:##:##:00
LS port     : 37c4f125-####-####-####-#########8c
urpf-mode   : STRICT_MODE                          >>>>>>>>>>>>>
admin       : up
op_state    : up
MTU         : 1476
arp_proxy   :
 
Edge1> get logical-router feca5512-####-####-####-#########89 forwarding
Mon Feb 12 2024 UTC 03:38:14.600
Logical Router
UUID                                   VRF    LR-ID  Name                              Type
feca5512-####-####-####-fd6a9bcac289   2      3      SR-tier0                          SERVICE_ROUTER_TIER0
IPv4 Forwarding Table
IP Prefix          Gateway IP                                Type        UUID                                   Gateway MAC
1.1.1.0/25         10.64.1.1                                route       61ca295f-####-####-####-##########be
1.1.2.0/24         10.64.1.1                                route       61ca295f-####-####-####-##########be
1.1.3.0/24                                                   route       e6807a42-####-####-####-##########ba
1.1.3.1/32                                                   route       73555c8d-####-####-####-##########72
Protocol-Unsupported Packet dropped due to known protocol like ARP, ICMP, DHCP but cannot be decoded completely No action unless there is traffic connection issue, such as no arp, no icmp reply and etc.
TTL-Exceeded Packet dropped due to TTL exceeding There is a L3 loop, check forwarding information of the complete setup
Kni Packet dropped due to lrouter port's companion KNI port failed to send Check cpu utilization of linux process such as L7 LB, this is due to linux side not able to dequeue the pkt fast enough.
# top
top - 03:18:57 up 8 days,  7:17,  1 user,  load average: 1.71, 2.20, 2.32
Tasks: 238 total,   3 running, 235 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.9 us, 13.6 sy,  0.0 ni, 77.9 id,  0.1 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem :  7842940 total,   136860 free,  5437616 used,  2268464 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2334760 avail Mem
 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+    TGID COMMAND
2784801 nsx-sha   20   0    3212   1004    916 S   0.0   0.0   0:00.00 2784801 sleep
2784770 root      20   0    8056   4100   3264 R   1.9   0.1   0:00.39 2784770 top
2784735 root      20   0    5168   4244   3504 S   0.0   0.1   0:00.04 2784735 bash
Rate-Limit Packet dropped due to ingress traffic rate higher than configured QOS (traffic contract)

Check if ingress traffic to the lrouter is much higher than the QOS config.
You can find the QOS configured using the below CLI,

Edge1> get logical-router 558ba208-####-####-####-##########ed qos-config
Mon Jan 29 2024 UTC 06:16:45.835
Logical Router
UUID           : 558ba208-####-####-####-##########ed
Direction      : Ingress
Rate (Mbps)    : 1
Burst (bytes)  : 1
IPsec Packet dropped during IPsec input processing

Aggregated Rx drop counter for IPsec related drop, further check IPsec tunnel stats for particular drop reason using following command

"get ipsecvpn tunnel stats <IPsec tunnel/VTI UUID>" 

edge1> get ipsecvpn tunnel stats cec70165-####-####-####-##########e3
Interface UID                      : ###
Interface UUID                     : cec70165-####-####-####-##########e3
VTI UUID                           : cec70165-####-####-####-##########e3
 
Stats
    Rx Pkts                            : 0             Tx Pkts                            : 0
    Rx Bytes                           : 0             Tx Bytes                           : 0
    Rx MSS Adjusted                    : 0             Tx MSS Adjusted                    : 0
    Rx MSS Ignored                     : 0             Tx MSS Ignored                     : 0
    Rx Drops                           : 0             Tx Drops                           : 0
    Rx Drop Crypto Failure             : 0             Tx Drop Crypto Failure             : 0
    Rx Drop Enqueue Failure            : 0             Tx Drop Enqueue Failure            : 0
    Rx Drop State Mismatch             : 0             Tx Drop State Mismatch             : 0
    Rx Drop Malformed                  : 0             Tx Drop Malformed                  : 0
    Rx Drop Proto Not Supported        : 0             Tx Drop Proto Not Supported        : 0
    Rx Drop Replay                     : 0             Tx Drop Seq Rollover               : 0
    Rx Drop Inner Malformed            : 0             Tx Drop Fragmentation Needed       : 0
    Rx Drop Policy Nomatch             : 0             Rekey Request Failure              : 0
    Rx Drop Auth Failure               : 0
    Rx Drop Zero Sequence Number       : 0
 
v6 Stats
    Rx Pkts                            : 3236          Tx Pkts                            : 3184
    Rx Bytes                           : 250276        Tx Bytes                           : 631712
    Rx MSS Adjusted                    : 0             Tx MSS Adjusted                    : 0
    Rx MSS Ignored                     : 0             Tx MSS Ignored                     : 0
    Rx Drops                           : 0             Tx Drops                           : 0
    Rx Drop Crypto Failure             : 0             Tx Drop Crypto Failure             : 0
    Rx Drop Enqueue Failure            : 0             Tx Drop Enqueue Failure            : 0
    Rx Drop State Mismatch             : 0             Tx Drop State Mismatch             : 0
    Rx Drop Malformed                  : 0             Tx Drop Malformed                  : 0
    Rx Drop Proto Not Supported        : 0             Tx Drop Proto Not Supported        : 0
    Rx Drop Replay                     : 0             Tx Drop Seq Rollover               : 0
    Rx Drop Inner Malformed            : 0             Tx Drop Fragmentation Needed       : 0
    Rx Drop Policy Nomatch             : 0             Rekey Request Failure              : 0
    Rx Drop Auth Failure               : 0
    Rx Drop Zero Sequence Number       : 0
----------------------------------------------------------------------------------------------------
IPsec-NoSA Packet dropped due to missing IPsec inbound or ingress security association SA

Check Outbound and Inbound SAs installed by running below commands

"get ipsecvpn sad" or

"get ipsecvpn sad summary"

edge1> get ipsecvpn sad summary
Outbound SAs (6):
----------------------------------------------------------------------------------------------------------------------------
 VRF   Rule ID      Src IP            Dest IP           Src Subnet          Dest Subnet         SPI          NAT   Rem Life
----------------------------------------------------------------------------------------------------------------------------
 2     1408217139   5050::100         2424::101                                                 0xc6a6b127    N    1642 sec
 2     334475315    5050::100         2424::101                                                 0xcbe1de7e    N    1581 sec
 2     536870913    1111::10          1111::20          192.168.1.0/24      192.168.5.0/24      0x2e733800    N     922 sec
 5     536870927    192.168.51.100    10.20.20.101      192.168.2.0/24      192.168.7.0/24      0xce11f2c1    N    2279 sec
 2     1610612738   1111::10          1111::20          fec0:1::/64         fec0:5::/64         0x2786ae00    N     880 sec
 5     1610612752   192.168.51.100    10.20.20.101      fec0:2::/64         fec0:7::/64         0xc02800f4    N    2279 sec
 
Inbound SAs (6):
----------------------------------------------------------------------------------------------------------------------------
 VRF   Rule ID      Src IP            Dest IP           Src Subnet          Dest Subnet         SPI          NAT   Rem Life
----------------------------------------------------------------------------------------------------------------------------
 5     2684354575   10.20.20.101      192.168.51.100    192.168.7.0/24      192.168.2.0/24      0xbf806100    N    2279 sec
 2     3555700787   2424::101         5050::100                                                 0xc3061f00    N    1642 sec
 5     3758096400   10.20.20.101      192.168.51.100    fec0:7::/64         fec0:2::/64         0xef28dd00    N    2279 sec
 2     2684354561   1111::20          1111::10          192.168.5.0/24      192.168.1.0/24      0x3c0ccb00    N     922 sec
 2     3758096386   1111::20          1111::10          fec0:5::/64         fec0:1::/64         0x23397400    N     880 sec
 2     2481958963   2424::101         5050::100                                                 0xe5eb0d00    N    1581 sec
IPsec-NoVTI Packet dropped due to IPsec input missing or admin-down VTI interface

VTI interface present on T0/T1 SR with mode as "vti" and we can find it by running

"get logical-router <T0/T1 SR UUID> interfaces"

edge1> get logical-router 220c92e1-####-####-####-##########7b interfaces
Logical Router
UUID                                   VRF    LR-ID  Name                              Type                      
220c92e1-####-####-####-#########7b   2      3      SR-ServerT0_AS                    SERVICE_ROUTER_TIER0      
Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable)
    Interface     : cec70165-####-####-####-#########e3
    Ifuid         : 295
    Mode          : vti
    Port-type     : vti
    IP/Mask       : 192.168.37.102/30;fec0:37::102/64(NA);fec0:37::50:56ff:fe01:400/64(NA);fe80::50:56ff:fe01:400/64(NA)
    Urpf-mode     : PORT_CHECK
    Admin         : up
    Op_state      : up




TX Interface Stats
 

Stats(Tx) Description Action
Blocked Packet dropped due to interface being blocked/admin-down due to SR of the edge node is in standby state See Rx
Firewall Packet dropped due to firewall See Rx
Frag-Needed Packet dropped due to the need of fragmentation DF bit is set but packet len is greater than than the MTU, check MTU configuration
No-neighbor Packet dropped due to ARP failure The neighbor IP has not reply the arp request, check the neighbor VM or router
No-Memory Packet dropped due to shortage of memory Potential mbuf/memory leaking
Edge1> get dataplane memory stats
Mon Feb 12 2024 UTC 03:39:39.950
Memory Usage
 
Available_entries             : 1024
Available_entries_in_cache    : 0
Cache_size_per_core           : 128
Name                          : jumbo_mbuf_pool
Size                          : 1024
No-Linked-Port Packet dropped due to failure in forwarding due to no linked peer lport

Check lrouter port configuration, it should have linked lswitch port like

nsx> get logical-router interface c32d0564-7####-####-####-#########ff       
Fri Jan 26 2024 PST 13:56:49.745
interface   : c32d0564-####-####-####-##########ff
LS port     : 18ba9b76-####-####-####-##########fb
Rate-Limit Packet dropped due to egress traffic rate higher than configured QOS (traffic contract)

Check if egress traffic to the lrouter is much higher than the QOS config.
You can find the QOS configured using the below CLI,

Edge1> get logical-router 558ba208-####-####-####-##########ed qos-config
Mon Jan 29 2024 UTC 06:16:45.835
Logical Router
UUID           : 558ba208-####-####-####-#########ed
Direction      : Egress
Rate (Mbps)    : 1
Burst (bytes)  : 1
IPsec Packet dropped during IPsec output processing

Aggregated Tx drop counter for IPsec related drop, further check IPsec tunnel stats for particular drop reason using following command

"get ipsecvpn tunnel stats <IPsec tunnel/VTI UUID>" 

edge1> get ipsecvpn tunnel stats cec70165-####-####-####-#########e3
Interface UID                      : ###
Interface UUID                     : cec70165-####-####-####-##########e3
VTI UUID                           : cec70165-####-####-####-##########e3
 
Stats
    Rx Pkts                            : 0             Tx Pkts                            : 0
    Rx Bytes                           : 0             Tx Bytes                           : 0
    Rx MSS Adjusted                    : 0             Tx MSS Adjusted                    : 0
    Rx MSS Ignored                     : 0             Tx MSS Ignored                     : 0
    Rx Drops                           : 0             Tx Drops                           : 0
    Rx Drop Crypto Failure             : 0             Tx Drop Crypto Failure             : 0
    Rx Drop Enqueue Failure            : 0             Tx Drop Enqueue Failure            : 0
    Rx Drop State Mismatch             : 0             Tx Drop State Mismatch             : 0
    Rx Drop Malformed                  : 0             Tx Drop Malformed                  : 0
    Rx Drop Proto Not Supported        : 0             Tx Drop Proto Not Supported        : 0
    Rx Drop Replay                     : 0             Tx Drop Seq Rollover               : 0
    Rx Drop Inner Malformed            : 0             Tx Drop Fragmentation Needed       : 0
    Rx Drop Policy Nomatch             : 0             Rekey Request Failure              : 0
    Rx Drop Auth Failure               : 0
    Rx Drop Zero Sequence Number       : 0
 
v6 Stats
    Rx Pkts                            : 3236          Tx Pkts                            : 3184
    Rx Bytes                           : 250276        Tx Bytes                           : 631712
    Rx MSS Adjusted                    : 0             Tx MSS Adjusted                    : 0
    Rx MSS Ignored                     : 0             Tx MSS Ignored                     : 0
    Rx Drops                           : 0             Tx Drops                           : 0
    Rx Drop Crypto Failure             : 0             Tx Drop Crypto Failure             : 0
    Rx Drop Enqueue Failure            : 0             Tx Drop Enqueue Failure            : 0
    Rx Drop State Mismatch             : 0             Tx Drop State Mismatch             : 0
    Rx Drop Malformed                  : 0             Tx Drop Malformed                  : 0
    Rx Drop Proto Not Supported        : 0             Tx Drop Proto Not Supported        : 0
    Rx Drop Replay                     : 0             Tx Drop Seq Rollover               : 0
    Rx Drop Inner Malformed            : 0             Tx Drop Fragmentation Needed       : 0
    Rx Drop Policy Nomatch             : 0             Rekey Request Failure              : 0
    Rx Drop Auth Failure               : 0
    Rx Drop Zero Sequence Number       : 0
----------------------------------------------------------------------------------------------------
IPsec-NoSA Packet dropped due to missing IPsec outbound or egress security association (SA)

Check Outbound and Inbound SAs installed by running below commands

"get ipsecvpn sad" or

"get ipsecvpn sad summary"

edge1> get ipsecvpn sad summary
Outbound SAs (6):
----------------------------------------------------------------------------------------------------------------------------
 VRF   Rule ID      Src IP            Dest IP           Src Subnet          Dest Subnet         SPI          NAT   Rem Life
----------------------------------------------------------------------------------------------------------------------------
 2     1408217139   5050::100         2424::101                                                 0xc6a6b127    N    1642 sec
 2     334475315    5050::100         2424::101                                                 0xcbe1de7e    N    1581 sec
 2     536870913    1111::10          1111::20          192.168.1.0/24      192.168.5.0/24      0x2e733800    N     922 sec
 5     536870927    192.168.51.100    10.0.0.1      192.168.2.0/24      192.168.7.0/24      0xce11f2c1    N    2279 sec
 2     1610612738   1111::10          1111::20          fec0:1::/64         fec0:5::/64         0x2786ae00    N     880 sec
 5     1610612752   192.168.51.100    10.0.0.1      fec0:2::/64         fec0:7::/64         0xc02800f4    N    2279 sec
 
Inbound SAs (6):
----------------------------------------------------------------------------------------------------------------------------
 VRF   Rule ID      Src IP            Dest IP           Src Subnet          Dest Subnet         SPI          NAT   Rem Life
----------------------------------------------------------------------------------------------------------------------------
 5     2684354575   10.0.0.1      192.168.51.100    192.168.7.0/24      192.168.2.0/24      0xbf806100    N    2279 sec
 2     3555700787   2424::101         5050::100                                                 0xc3061f00    N    1642 sec
 5     3758096400   10.0.0.1      192.168.51.100    fec0:7::/64         fec0:2::/64         0xef28dd00    N    2279 sec
 2     2684354561   1111::20          1111::10          192.168.5.0/24      192.168.1.0/24      0x3c0ccb00    N     922 sec
 2     3758096386   1111::20          1111::10          fec0:5::/64         fec0:1::/64         0x23397400    N     880 sec
 2     2481958963   2424::101         5050::100                                                 0xe5eb0d00    N    1581 sec
IPsec-NoVTI Packet dropped due to IPsec input missing or down VTI interface

VTI interface present on T0/T1 SR with mode as "vti" and we can find it by running

"get logical-router <T0/T1 SR UUID> interfaces"

edge1> get logical-router 220c92e1-####-####-####-#########7b interfaces
Logical Router
UUID                                   VRF    LR-ID  Name                              Type                      
220c92e1-####-####-####-#########7b   2      3      SR-ServerT0_AS                    SERVICE_ROUTER_TIER0      
Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable)
    Interface     : cec70165-####-####-####-#########e3
    Ifuid         : ###
    Mode          : vti
    Port-type     : vti
    IP/Mask       : 192.168.37.102/30;fec0:37::102/64(NA);fec0:37::50:56ff:fe01:400/64(NA);fe80::50:56ff:fe01:400/64(NA)
    Urpf-mode     : PORT_CHECK
    Admin         : up
    Op_state      : up
IPsec-Policy-Error Packet dropped due to missing rule for IPsec processing. SPD look-up failed.

Check below command output there should not be any entry with tag "ipsec"

edge1> get firewall ded3f395-####-####-####-###########37 ike policy
IKE policy count: 4
    Rule ID   : 536870913
    Policy    : out protocol any stateless from ip 192.168.1.0/24 to ip 192.168.5.0/24 secure keypolicy 00003400-2000-0000-2000-000100000000 tag 'ipsec'
 
    Rule ID   : 1610612738
    Policy    : out protocol any stateless from ip fec0:1::/64 to ip fec0:5::/64 secure keypolicy 00003400-2000-0000-2000-000200000000 tag 'ipsec'
IPsec-Policy-Block Packet dropped due to IPsec rule with DROP as action

Check below command output for tag "DROP"

edge1> get firewall ded3f395-####-####-####-##########37 ike policy
IKE policy count: 4
    Rule ID   : 536870913
    Policy    : out protocol any stateless from ip 192.168.1.0/24 to ip 192.168.5.0/24 secure keypolicy 00003400-2000-0000-2000-000100000000 tag 'ipsec'
 
    Rule ID   : 1610612738
    Policy    : out protocol any stateless from ip fec0:1::/64 to ip fec0:5::/64 secure keypolicy 00003400-2000-0000-2000-000200000000 tag 'ipsec'

 

Powered by Wolken Software