Troubleshooting NSX-T using Packet Captures
search cancel

Troubleshooting NSX-T using Packet Captures

book

Article ID: 345925

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

This KB is for troubleshooting NSX-T using packet captures. This will help us identifying the traffic drop on virtual components including East-West and North-South Traffic.

Environment

VMware NSX-T Data Center

Resolution

(A). FOR EAST-WEST TRAFFIC:

 

Perform the following steps in both the Source and destination ESXi host.

 

  • Figure the switchport and uplink, where traffic flows for the VM

 

1. Login to ESxi hosts where the affected vm sits

2. Commands to pull Switch port info:

 

esxcli network vm list  --> (Copy the world ID and place it in port number section below)

esxcli network vm port list -w <port number>

 

"Port ID:" and "Team Uplink:" will give the respective switchport and uplink.

 

 

  • Once the Switchport and vmnic is figured, use the below commands for captures:

 

[To see live traffic]

 

pktcap-uw --switchport <switchport-id> --capture VnicTx,VnicRx -o- | tcpdump-uw -r - -nne

 

pktcap-uw --uplink <vmnic_number> --capture UplinkSndKernel,UplinkRcvKernel   -o- | tcpdump-uw -r - -nne 

 

Ctrl+C to stop the capture

Note:
UplinkRcvKernel -- The function that receives packets from uplink dev at kernel side
UplinkSndKernel -- Function to Tx packets on uplink at kernel side
VnicTx -- Function in vnic backend to Tx packets from guest
VnicRx -- Function in vnic backend to Rx packets to guest
 

[To capture and write on pcap]

 

pktcap-uw --switchport <switchport-id> --dir 2 -o /vmfs/volumes/<datastore_name>/switchport _capture.pcap  

 

pktcap-uw --uplink <vmnic_no> --capture UplinkSndKernel -o /vmfs/volumes/<datastore_name>/UplinkSndKernel-vmnic-esxi.pcap & pktcap-uw --uplink <vmnic_no> --capture UplinkRcvKernel -o /vmfs/volumes/<datastore_name>/UplinkRcdKernel-vmnic-esxi.pcap


command to kill capture process:

kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)

 

(NOTE: Change the vmnic & IP as applicable based on situation and VM.)


sample:

Source IP: 10.0.0.10
Destination IP: 20.0.0.10
Source ESXi: esx-04
Destination ESXi: esx-05

[root@esx-04:~] esxcli network vm list | grep -i VM-1
  287042  VM-1          1

[root@esx-04:~] esxcli network vm list
World ID  Name  Num Ports  Networks
--------  ----  ---------  --------
  287042  VM-1          1

[root@esx-04:~] esxcli network vm port list -w 287042
   Port ID: 67108880
   vSwitch: RegionA01-VDS7
   Portgroup:
   DVPort ID: 5ea27e9d-10e0-49bb-84e1-bc87ceab5383
   MAC Address: 00:50:56:9d:04:b9
   IP Address: 0.0.0.0
   Team Uplink: vmnic0
   Uplink Port ID: 2214592517
   Active Filters: vmware-sfw

[root@esx-04:~] pktcap-uw --switchport 67108880 --capture VnicTx,VnicRx -o- | tcpdump-uw -r - -nne
The switch port id is 0x04000010.
The session capture point is VnicTx,VnicRx.
pktcap: The output file is -.
pktcap: No server port specifed, select 27887 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 27887.
pktcap: Main thread: 287048629056.
pktcap: Dump Thread: 287049164544.
pktcap: Recv Thread: 287049692928.
pktcap: Accept...
pktcap: Vsock connection from port 1028 cid 2.
reading from file -, link-type EN10MB (Ethernet)
05:28:17.947352 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.947733 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64
05:28:18.947514 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.948003 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
05:28:19.947650 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1028, length 64
05:28:19.948031 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1028, length 64
05:28:20.947780 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1029, length 64
05:28:20.948260 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1029, length 64
05:28:21.947919 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1030, length 64
05:28:21.948351 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1030, length 64
05:28:22.948016 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1031, length 64
05:28:22.948465 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1031, length 64
05:28:23.948117 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1032, length 64
05:28:23.948502 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1032, length 64
05:28:24.948349 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1033, length 64
05:28:24.948730 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1033, length 64
05:28:25.948447 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1034, length 64
05:28:25.948792 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1034, length 64
05:28:26.948542 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1035, length 64
05:28:26.948903 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1035, length 64
05:28:27.948639 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1036, length 64
05:28:27.949031 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1036, length 64
05:28:28.948741 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1037, length 64
05:28:28.949159 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1037, length 64
05:28:29.948847 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1038, length 64
05:28:29.949365 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1038, length 64
05:28:30.948944 00:50:56:9d:04:b9 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1039, length 64
pktcap: Thread 287048629056 received signal 2.
tcpdump-uw: pcap_loop: error reading dump file: Interrupted system call
pktcap: Join with dump thread failed.
pktcap: Destroying session 2.
pktcap:
pktcap: Dumped 50 packet to file -, dropped 0 packets.
pktcap: Done.

[root@esx-04:~] pktcap-uw --uplink vmnic0 --capture UplinkSndKernel,UplinkRcvKernel -o- | tcpdump-uw -r - -nne | grep -i 20.0.0.10
The name of the uplink is vmnic0.
The session capture point is UplinkSndKernel,UplinkRcvKernel.
pktcap: The output file is -.
pktcap: No server port specifed, select 27896 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 27896.
pktcap: Main thread: 419002739520.
pktcap: Dump Thread: 419003275008.
reading from file -, link-type EN10MB (Ethernet)
pktcap: Recv Thread: 419003803392.
pktcap: Accept...
pktcap: Vsock connection from port 1029 cid 2.
05:28:17.947409 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.947716 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64
05:28:18.947600 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.947968 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
05:28:19.947699 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1028, length 64
05:28:19.948018 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1028, length 64
05:28:20.947844 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1029, length 64
05:28:20.948232 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1029, length 64
05:28:21.947989 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1030, length 64
05:28:21.948319 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1030, length 64


[root@esx-05:~] esxcli network vm list | grep -i VM-2
  286924  VM-2          1

[root@esx-05:~] esxcli network vm port list -w 286924
   Port ID: 67108880
   vSwitch: RegionA01-VDS7
   Portgroup:
   DVPort ID: c14db2a2-c202-4c1c-bf8f-a702589a834a
   MAC Address: 00:50:56:9d:0c:73
   IP Address: 0.0.0.0
   Team Uplink: vmnic0
   Uplink Port ID: 2214592517
   Active Filters: vmware-sfw



[root@esx-05:~] pktcap-uw --uplink vmnic0 --capture UplinkSndKernel,UplinkRcvKernel -o- | tcpdump-uw -r - -nne | grep -i 20.0.0.10
The name of the uplink is vmnic0.
The session capture point is UplinkSndKernel,UplinkRcvKernel.
pktcap: The output file is -.
pktcap: No server port specifed, select 28035 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 28035.
pktcap: Main thread: 603350494016.
pktcap: Dump Thread: 603351029504.
reading from file -, link-type EN10MB (Ethernet)
pktcap: Recv Thread: 603351557888.
pktcap: Accept...
pktcap: Vsock connection from port 1031 cid 2.
05:28:17.948445 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.948565 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64
05:28:18.948662 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.948803 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
05:28:19.948725 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1028, length 64
05:28:19.948849 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1028, length 64
05:28:20.948890 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1029, length 64
05:28:20.949038 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1029, length 64
05:28:21.949022 00:50:56:60:17:49 > 00:50:56:61:25:c6, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1030, length 64
05:28:21.949152 00:50:56:61:25:c6 > 00:50:56:60:17:49, ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: 02:50:56:56:44:52 > 00:50:56:9d:04:b9, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1030, length 64

[root@esx-05:~] pktcap-uw --switchport 67108880 --capture VnicTx,VnicRx -o- | tcpdump-uw -r - -nne
The switch port id is 0x04000010.
The session capture point is VnicTx,VnicRx.
pktcap: The output file is -.
pktcap: No server port specifed, select 28025 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 28025.
pktcap: Main thread: 748447206208.
pktcap: Recv Thread: 748448270080.
pktcap: Accept...
pktcap: Vsock connection from port 1030 cid 2.
pktcap: Dump Thread: 748447741696.
reading from file -, link-type EN10MB (Ethernet)
05:28:17.948464 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.948531 00:50:56:9d:0c:73 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64

05:28:18.948685 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.948769 00:50:56:9d:0c:73 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
05:28:19.948747 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1028, length 64
05:28:19.948815 00:50:56:9d:0c:73 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1028, length 64
05:28:20.948912 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1029, length 64
05:28:20.949004 00:50:56:9d:0c:73 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1029, length 64
05:28:21.949047 02:50:56:56:44:52 > 00:50:56:9d:0c:73, ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 20.0.0.10: ICMP echo request, id 32771, seq 1030, length 64
05:28:21.949114 00:50:56:9d:0c:73 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 20.0.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1030, length 64
pktcap: Thread 748447206208 received signal 2.
tcpdump-uw: pcap_loop: error reading dump file: Interrupted system call
pktcap: Join with dump thread failed.
pktcap: Destroying session 4.
pktcap:
pktcap: Dumped 25 packet to file -, dropped 0 packets.
pktcap: Done.


(B). FOR NORTH-SOUTH TRAFFIC:

 

  • To capture traffic on ESXi end, the same is applied as stated in "EAST WEST TRAFFIC" section.

 

  • To capture on the Edge Uplink:

 

1. Login to the active Edge Node

2. get into the vrf of SERVICE_ROUTER_TIER0

 

 get logical-router    <<<<< list of logical-router

 vrf <vrf_number>     <<<<< entering the respective vrf

 get interface      <<<<< list all the interfaces and look for "Port-type: uplink" "Interface" "IP" "MAC" to get the Edge uplink interface details

 exit

 

sample:

 

edge01> get logical-router

Fri Mar 17 2023 UTC 06:21:39.455

Logical Router

UUID                  VRF  LR-ID Name               Type            Ports  Neighbors

736a80e3-23f6-5a2d-81d6-bbefb2786666  0   0                    TUNNEL           4    6/5000

7d50e317-bec3-4d50-91b5-37cb1a28990e  4   16   DR-T1-Gateway-01         DISTRIBUTED_ROUTER_TIER1  8    2/50000

b9cac14c-4c9a-4331-8e0b-40119f821cfb  5   1   DR-T0-Gateway-01         DISTRIBUTED_ROUTER_TIER0  5    2/50000

c4dce003-312e-4c29-8f2a-42326b223dee  6   17   SR-T0-Gateway-01         SERVICE_ROUTER_TIER0    6    2/50000 <<<

 

edge01> vrf 6

edge01(tier0_sr[6])>

 

edge01(tier0_sr[6])> get interface

 

edge01(tier0_sr[6])> get interface

Fri Mar 17 2023 UTC 06:24:10.473

Logical Router

UUID                  VRF  LR-ID Name               Type

c4dce003-312e-4c29-8f2a-42326b223dee  6   17   SR-T0-Gateway-01         SERVICE_ROUTER_TIER0

Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable)

  Interface   : f65978d0-73ba-480b-b984-ec05e4b42a40 <<<<<<

  Ifuid     : 325

  Name     : EdgeUplinkB-TN1

  Fwd-mode   : IPV4_AND_IPV6

  Internal name : uplink-325

  Mode     : lif

  Port-type   : uplink <<<<<<<<

  IP/Mask    : 192.168.133.1/24 <<<<<<<<

  MAC      : 00:50:56:9d:9c:c5 <<<<<<<

  VLAN     : 133

  Access-VLAN  : untagged

  LS port    : e87abfd3-c2da-4e20-a106-2babd1d89cd6

  Urpf-mode   : STRICT_MODE

  DAD-mode   : LOOSE

  RA-mode    : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)

  Admin     : up

  Op_state   : up

  Enable-mcast : False

  MTU      : 8800

  arp_proxy   :

 

edge01(tier0_sr[6])> exit

 

 

3. Captures

 

[To see live traffic]

 

start capture interface <port-uuid-name> direction dual expression <expression>

 

Ctrl + C to stop capture

 

Sample:

 

edge01> start capture interface f65978d0-73ba-480b-b984-ec05e4b42a40 direction dual expression host 192.168.133.1

06:30:37.592896 00:50:56:9d:9c:c5 > 00:50:56:01:06:f8, ethertype IPv4 (0x0800), length 66: 192.168.133.1.56308 > 192.168.133.254.3784: BFDv1, Control, State Up, Flags: [none], length: 24

<base64>AFBWAQb4AFBWnZzFCABFwAA0AAAAAP8RLqjAqIUBwKiF/tv0DsgAIK3iIMADGCF4DxuO43LQAA9CQAAPQkAAAAAA</base64>

 

06:30:37.873333 00:50:56:01:06:f8 > 00:50:56:9d:9c:c5, ethertype IPv4 (0x0800), length 66: 192.168.133.254.49180 > 192.168.133.1.3784: BFDv1, Control, State Up, Flags: [none], length: 24

<base64>AFBWnZzFAFBWAQb4CABFwAA0IpBAAP8RzBfAqIX+wKiFAcAcDsgAIAZqIMADGI7jctAheA8bAA9CQAAPQkAAAMNQ</base64>

 

06:30:38.433363 00:50:56:9d:9c:c5 > 00:50:56:01:06:f8, ethertype IPv4 (0x0800), length 66: 192.168.133.1.56308 > 192.168.133.254.3784: BFDv1, Control, State Up, Flags: [none], length: 24

<base64>AFBWAQb4AFBWnZzFCABFwAA0AAAAAP8RLqjAqIUBwKiF/tv0DsgAIK3iIMADGCF4DxuO43LQAA9CQAAPQkAAAAAA</base64>

 

06:30:38.743828 00:50:56:01:06:f8 > 00:50:56:9d:9c:c5, ethertype IPv4 (0x0800), length 66: 192.168.133.254.49180 > 192.168.133.1.3784: BFDv1, Control, State Up, Flags: [none], length: 24

<base64>AFBWnZzFAFBWAQb4CABFwAA0JBFAAP8RypbAqIX+wKiFAcAcDsgAIAZqIMADGI7jctAheA8bAA9CQAAPQkAAAMNQ</base64>

 

06:30:39.382513 00:50:56:9d:9c:c5 > 00:50:56:01:06:f8, ethertype IPv4 (0x0800), length 66: 192.168.133.1.56308 > 192.168.133.254.3784: BFDv1, Control, State Up, Flags: [none], length: 24

<base64>AFBWAQb4AFBWnZzFCABFwAA0AAAAAP8RLqjAqIUBwKiF/tv0DsgAIK3iIMADGCF4DxuO43LQAA9CQAAPQkAAAAAA</base64>

 

^C

5 packets captured

5 packets received by filter

0 packets dropped by kernel

 

[To capture and write on pcap]

  • define capture session

set capture session <session-number> interface <port-uuid-name> direction dual

  • View capture session

get capture session

  • Start capture session

set capture session <session-number> file <filename> expression <expression>

  • View capture files

get files

  • Copy capture files

copy file <filename> url scp://username@ip_address/filepath/filename

Alternatively, you can just scp using any client (WinSCP) etc to the NSX-T edge node and extract the files from stored directory.

Note: generated pcaps are stored in "/image/vmware/nsx/file-store/" directory on NSX-T edge node.


sample:

 

edge01> set capture session 0 interface f65978d0-73ba-480b-b984-ec05e4b42a40 direction dual

 

edge01> get capture session

Fri Mar 17 2023 UTC 06:40:03.244

Packet Capture Session

ID          : 0

PORTS        : ['f65978d0-73ba-480b-b984-ec05e4b42a40']

 

Packet Capture Session

ID          : 1

PORTS        : []

 

Packet Capture Session

ID          : 2

PORTS        : []

 

Packet Capture Session

ID          : 3

PORTS        : []

 

Packet Capture Session

ID          : 4

PORTS        : []

 

Packet Capture Session

ID          : 5

PORTS        : []

 

edge01> set capture session 0 file Test_Capture.pcap expression host 192.168.133.254

 

Capture to file initiated, enter Ctrl-C to terminate

 

^C11 packets captured

12 packets received by filter

0 packets dropped by kernel

 

edge01> get files

Fri Mar 17 2023 UTC 06:43:04.413

Directory of filestore:/

 

    -rw-    5398   Nov 03 2022 23:52:31 UTC nsx_backup_cleaner.py

    -rw-    9967   Nov 03 2022 23:52:31 UTC backup_restore_helper.py

    -rw-     972   Mar 17 2023 06:42:42 UTC Test_Capture.pcap <<<<<<<<<<

    -rw-    5748   Nov 03 2022 23:52:31 UTC get_backup_timestamps.sh

 

 

edge01> copy file Test_Capture.pcap url scp://[email protected]/tmp/

Are you sure you want to continue connecting (yes/no)? yes

Password:

Test_Capture.pcap               100% 972   1.2MB/s  00:00

edge01>