Discrepancy Between Application Script Performance and Raw Network Throughput on NSX Overlay Segments

Products

• VMware NSX

Issue/Introduction

• Application teams report severe throughput degradation (e.g., ~35 Mbps) between VMs across different sites or segments (e.g., DMZ to Database).
• The performance issue is identified using custom application-layer scripts (e.g., PostgreSQL latency scripts or file transfer tasks).
• Infrastructure logs (ESXi, NSX Edge, and Physical Switches) show no packet drops, CRC errors, or hardware anomalies.
• Traffic within the same site/segment (East-West) performs at expected speeds.

Environment

VMware NSX 

Resolution

To determine the true health of the network datapath, you must isolate the network layer from the application layer using standardized benchmarking.

1. Perform a Raw Network Benchmark (iperf3)

Discontinue the use of application-layer scripts for baseline validation. Use iperf3 to measure raw TCP/UDP throughput.

Server Side (Receiver VM):

Note: Ensure TCP port 5201 is allowed on the Distributed Firewall (DFW) and any physical firewalls in the path.

Client Side (Sender VM): Run the test using parallel streams to simulate real-world multi-connection application behavior:

-P 8: Initiates 8 parallel streams to utilize multiple CPU cores and network queues.
-t 60: Runs the test for a duration long enough to observe stability.

2. Analyze the Discrepancy

Network Healthy: If iperf3 shows high throughput (e.g., 400+ Mbps or near line rate) while the application script remains at ~35 Mbps, the issue is not the NSX network. Redirect troubleshooting to application tuning or VM CPU/Memory contention.
Network Bottleneck: If iperf3 also shows low throughput, check for physical MTU mismatches (Jumbo Frames), Edge CPU saturation, or physical firewall rate-limiting.

3. Verification of Path Integrity

If throughput is inconsistent, perform a traceroute to confirm if traffic is hairpining through an unexpected Edge node or physical firewall: