Troubleshooting Port Mirroring: "Encapsulated Remote Mirroring (L3) Source" and "Remote L3 Span"

Products

• VMware vSphere ESXi
• VMware vCenter Server
• VMware NSX

Issue/Introduction

• A Port Mirroring Session is enabled in a vSphere Distributed Switch (vDS)/NSX for one or more VMs.
• The mirroring type is "Encapsulated Remote Mirroring (L3) Source" or "Remote L3 Span."
• The user is unable to see the mirrored traffic on the target monitoring server.

Environment

VMware vSphere ESXi.

VMware vCenter Server.

VMware NSX.

Resolution

• Ensure the ESXi hosts DO NOT show as out of sync with the distributed switch.
• Compare the port numbers listed in the vDS and confirm they are the same on the hosts using the command "esxcfg-vswitch -l".
• Ensure the management vmk adapter can reach the target server with vmkping, as mirrored packets are forwarded through the management network by default. (vmkping -I <vmk#> <Target_IP>).
• Perform a packet capture to validate whether the host is sending the GRE traffic to the target.
  
  

  [root@esxi:] pktcap-uw --vmk <vmk#> --ip <TARGET_IP> --dir 2 -o -|tcpdump-uw -enr -
  The name of the vmk is vmk0.
  The session filter IP(src or dst) address is ##.##.##.33.
  pktcap: The output file is -.
  pktcap: No server port specifed, select 49178 as the port.
  pktcap: Local CID 2.
  pktcap: Listen on port 49178.
  pktcap: Main thread: 790887431360.
  pktcap: Dump Thread: 790887962368.
  pktcap: Recv Thread: 790888490752.
  pktcap: Accept...
  pktcap: The output file format is pcapng.
  pktcap: Vsock connection from port 1027 cid 2.
  reading from file -, link-type EN10MB (Ethernet), snapshot length 65535
  03:35:42.537760 00:##:##:##:##:e7 > 00:##:##:##:##:1b, ethertype IPv4 (0x0800), length 171: ##.##.##.32 > ##.##.##.33: GREv0, seq 1315, proto unknown (0x88be), length 137: gre-proto-0x88be
  03:35:42.537782 00:##:##:##:##:e7 > 00:##:##:##:##:1b, ethertype IPv4 (0x0800), length 188: ##.##.##.32 > ##.##.##.33: GREv0, seq 1316, proto unknown (0x88be), length 154: gre-proto-0x88be
  03:35:42.537783 00:##:##:##:##:e7 > 00:##:##:##:##:1b, ethertype IPv4 (0x0800), length 176: ##.##.##.32 > ##.##.##.33: GREv0, seq 1317, proto unknown (0x88be), length 142: gre-proto-0x88be
  03:35:42.537784 00:##:##:##:##:e7 > 00:##:##:##:##:1b, ethertype IPv4 (0x0800), length 124: ##.##.##.32 > ##.##.##.33: GREv0, seq 1318, proto unknown (0x88be), length 90: gre-proto-0x88be

   

   Note: You won't see this traffic generated if the target server is unreachable by the desired vmk adapter.

• Engage the physical network team if it is confirmed that the mirrored GRE traffic is departing from the active physical NIC.

Additional Information

ESXi does have an option to create a custom netstack vmk adapter for mirrored traffic through the CLI (e.g., esxcli network ip netstack add -N="mirror").

If such an adapter is used and assigned, it will replace the management adapter and must be reachable to the target.

Create a Port Mirroring Session in VMware vSphere.
Add a Port Mirroring Session in VMware NSX.