In an NSX-T environment configured for EVPN, traffic is dropped due to packet size

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX-T Data Center versions up to and including NSX-T 3.1.2.1
• EVPN feature is configured and is in use
• Traffic is dropped on the Edge VXLAN tunnel interface
• Packet capture may show ICMP messages indicating fragmentation is needed

ICMP 10.10.10.1 unreachable - need to frag (mtu 1438)

• On the Edge admin cli, the VXLAN tunnel shows that the MTU is 1500 and traffic is being dropped due to a requirement to fragment

  get tunnel-ports | json
   {
    "admin": "down",
    "encap": "VXLAN",
    "ifuid": 507,
    "ifuuid": "9b431add-af4f-51fb-8c21-a151468fcf87",
    "last-update-mtu": "2021-06-21 14:45:53.739",
    "local-vtep-ip": "10.10.11.1",
    "local_vtep_label": 0,
    "lrouter_port": "7c535fd2-6ab3-52f9-b48f-c3ef5452ccf4",
    "mtu": 1500, <<<<<<<<
    "remote-vtep-ip": "112.1.1.100",
    "stats": {
            "rx_pkts": 87348267,
            "rx_bytes": 26867142436,
            "rx_drops": 0,
            "tx_pkts": 126698884,
            "tx_bytes": 61507144578,
            "tx_drops": 248332,
            "rx_drop_no_match": 0,
            "rx_drop_malformed": 0,
            "rx_drop_l2_loop": 0,
            "rx_drop_ra_guard": 0,
            "rx_drop_blocked": 0,
            "tx_drop_blocked": 0,
            "tx_drop_no_mem": 0,
            "tx_drop_no_route": 0,
            "tx_drop_no_nexthop": 0,
            "tx_drop_frag_needed": 248332 <<<<<<
    },
    "type": "vtep"
  }

Environment

VMware NSX-T Data Center 2.x
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center
VMware NSX-T

Cause

The VXLAN tunnel MTU by default should not be set.
The tunnel then learns the MTU using PMTUD from the the ICMP messages received.
In this problem scenario, the tunnel incorrectly has a default MTU of 1500 configured on the VXLAN tunnel.
This setting has the affect of disabling MTU learning.
As a result traffic which has a packet size too large for the datapath and has the do-not-fragment bit enabled will be dropped.

Resolution

This is a known issue affecting NSX-T Datacenter.

Workaround:
To workaround the issue, the Global MTU value can be configured to a value that allows traffic to pass

Networking -> Global Networking Config -> Gateway Interface MTU

Verify the setting has taken effect using the Edge admin cli get tunnel-ports | json from the Edge admin cli.