Troubleshooting DNS Connectivity from NSX-T VMs to Physical DNS Servers.

Products

• VMware NSX

Issue/Introduction

• VMs on NSX-T Overlay segments cannot resolve hostnames via DNS servers on the physical network.
• Infrastructure: Tier-0/Tier-1 Gateway configured with Source NAT (SNAT)
• Standard ICMP (Ping) tests fail or are disabled on the target DNS server.
• Application/TCP-based connectivity tests (e.g., Telnet on port 53) fail to establish a session.
• If ICMP is disabled, use a TCP-based check to verify Layer 4 connectivity. On a Windows VM, run: telnet <DNS_SERVER_IP> 53
• To determine if traffic is reaching the Edge or being dropped/redirected, perform a capture on the relevant Edge Node by running the following command :
  • start capture interface <INTERFACE_ID> direction dual expression host <DNS_SERVER_IP> and tcp port 53
• On the associated Edge's capture, the following was seen:

<timestamp> 00:50:56:##.##.## > b4:0c:25:##.##.##, ethertype IPv4 (0x0800), length 66: 172.##.##.##.49687 > 10.##.##.##.53: Flags [SEW], seq 3508334278, win 62622, options [mss 8946,nop,wscale 8,nop,nop,sackOK], length 0
<timestamp> 00:50:56:##.##.## > b4:0c:25:##.##.##, ethertype IPv4 (0x0800), length 66: 172.##.##.##.49687 > 10.##.##.##.53: Flags [S], seq 3508334278, win 62622, options [mss 8946,nop,wscale 8,nop,nop,sackOK], length 0

• Another capture was performed on the ESXi Host hosting the associated the Edge Node to verify if the NAT translation was occurring before the packets hit the physical side :

<timestamp> 00:50:56:##.##.## > b4:0c:25:##.##.##, ethertype IPv4 (0x0800), length 66: 10.##.##.##.61443 > 10.##.##.##.53: Flags [SEW], seq 794851963, win 62622, options [mss 8946,nop,wscale 8,nop,nop,sackOK], length 0
<timestamp> 00:50:56:##.##.## > b4:0c:25:##.##.##, ethertype IPv4 (0x0800), length 66: 10.##.##.##.61443 > 10.##.##.##.53: Flags [S], seq 794851963, win 62622, options [mss 8946,nop,wscale 8,nop,nop,sackOK], length 0

Environment

VMware NSX

Cause

• The Edge receives the packet from the internal IP 172.##.##.##. Note the repetitive [S] (SYN) flags without a corresponding [SA] (SYN-ACK).
• On the ESXi Host,  there is a confirmation that the Source NAT is successful. The source IP has been changed from 172.##.##.## to 10.##.##.##. The packet is leaving the NSX-T fabric, but the physical side is not responding.

Resolution

• The issue was caused by a missing return route on the physical network. While the physical infrastructure could reach the VM's original subnet, it had no routing table entry for the SNAT IP/Pool (10.##.##.##).
• Consequently, the DNS server's response packets were being dropped by the physical gateway as "unroutable."