High latency to local gateway for ICMP traffic observed on NSX overlay segments.

Products

• VMware NSX

Issue/Introduction

• VMs connected to overlay segments within NSX see intermittent high latency to their local gateway for ICMP requests.
• This can impact any ICMP traffic (such as ping and traceroute) targeting the gateway IP.
• Traceroutes may show intermittent high latency on the first hop (the local gateway on the T1/T0) but hops beyond the first hop will have lower latencies.

Environment

VMware NSX
VMware NSX-T Data Center

Cause

VMs connected to overlay segments will normally use T1 or T0 as their local gateway. When using the local gateway the VM will be pinging a local instance of the Virtual Distributed Router (VDR) within their local host. This VDR does not respond to ICMP echo request in a prioritized fashion and instead pushes the reply to a queue which then gets answered in the next running of that queue.  This means traffic going through the VDR will be processed with priority but ICMP targeting the VDR itself will appear to have high latency.

This can be seen for instance when tracing from a VM to an external northbound peer. The first local gateway hop may show higher latency than the further hops, even though the first hop is being passed through for subsequent traffic. This demonstrates it is the local gateway delaying its response to its own ICMP traffic and not all traffic being handled by it.

Resolution

This is expected behaviour and is not indicative of an issue. As long as traffic being forwarded through the VDR is not showing delays then the responses from the VDR itself can be ignored.