Troubleshooting SD-WAN path packet loss if underlay ping result without loss found

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

Packet loss is reported on VCO Monitor/Paths page. Ping test has already performed on underlay network but there is no loss found.

Environment

VeloCloud SD-WAN all supported releases

Cause

Path packet loss is monitored based on VCMP packets which is transported on UDP port 2426. It is possible that UDP 2426 packets are dropped by QoS or security policies on the network along the path, while ICMP packets are not.

Resolution

For instance, packet loss is reported on the path from edge1 GE3 192.168.1.1 to edge2 GE4 192.168.2.1.

• Run packet capture at the same time on both path peers regarding UDP2426 packets and save it to pcap file.
  Example:

vctcpdump -enni ge3 host 192.168.2.1 and port 2426 -s 100 -w /tmp/edge1_ge3.pcap
vctcpdump -enni ge3 host 192.168.1.1 and port 2426 -s 100 -w /tmp/edge2_ge4.pcap

 

• Open captured file by Wireshark. Under menu Analyze > Decode As, Add UDP 2426 to be decoded as UDPENCAP.

• Then ESP sequence number is available to see.

 

• Apply the same display filter on both peers' capture files.
  1 Specify the ip.src or ip.dst to filter the packets with same direction
  2 Specify the lowest and highest esp.sequence to filter the same packets range that both peers should have.
  
  Example:

ip.src==192.168.1.1 and esp.sequence >=15875065 and esp.sequence <=16122437

 

• Compare the displayed number on Wireshark bottom, if there is difference then it indicates the loss.

• If receiving peer's ESP sequence is out of order, this will also cause path packet loss to be accounted.