Symptoms:
The VXLAN tunnel MTU by default should not be set.
The tunnel then learns the MTU using PMTUD from the the ICMP messages received.
In this problem scenario, the tunnel incorrectly has a default MTU of 1500 configured on the VXLAN tunnel.
This setting has the effect of disabling MTU learning.
As a result, traffic which has a packet size too large for the data path and has the do-not-fragment bit enabled will be dropped.
This is a known issue affecting NSX-T Datacenter.
Workaround:
To workaround the issue, the Global MTU value can be configured to a value that allows traffic to pass.
Networking -> Global Networking Config -> Gateway Interface MTU
Verify the setting has taken effect using the Edge admin cli get tunnel-ports | json from the Edge admin CLI.