Troubleshooting VMware NSX-T using Packet Captures
Article ID: 345925
Updated On:
Products
Issue/Introduction
Environment
Resolution
For East-West Traffic
Perform the following steps in both the Source and destination of the VMware ESXi host.
- Identify the switchport and uplink, where traffic flows for the VM.
- Login to ESXi hosts where the affected VM is configured.
- Commands to pull Switch port info:esxcli network vm list -->
(Copy the world ID and place it in port number section below)
esxcli network vm port list -w <port number>"Port ID:" and "Team Uplink:" gives the respective switchport and uplink.
- After the Switchport and vmnic is identified, use the below commands for captures:
To see live traffic
- UplinkRcvKernel -- The function that receives packets from uplink dev at kernel side
- UplinkSndKernel -- Function to Tx packets on uplink at kernel side
- VnicTx -- Function in vnic backend to Tx packets from guest
- VnicRx -- Function in vnic backend to Rx packets to guest
To capture and write on pcap
To Kill the Capture Process
Note: Change the vmnic & IP as applicable based on situation and VM.
Sample:
Source IP: 10.0.0.10
Destination IP: 10.1.0.10
Source ESXi: esx-04
Destination ESXi: esx-05
[root@esx-04:~] esxcli network vm list | grep -i VM-1
287042 VM-1 1
[root@esx-04:~] esxcli network vm list
World ID Name Num Ports Networks
-------- ---- --------- --------
287042 VM-1 1
[root@esx-04:~] esxcli network vm port list -w 287042
Port ID: <port ID>
vSwitch: RegionA01-VDS7
Portgroup:
DVPort ID: 5ea27e9d-10e0-49bb-84e1-bc87ceab5383
MAC Address: MAC_ADDR_1
IP Address: 0.0.0.0
Team Uplink: vmnic0
Uplink Port ID: <uplink port ID>
Active Filters: vmware-sfw
[root@esx-04:~] pktcap-uw --switchport 67108880 --capture VnicTx,VnicRx -o- | tcpdump-uw -r - -nne
The switch port id is 0x04000010.
The session capture point is VnicTx,VnicRx.
pktcap: The output file is -.
pktcap: No server port specifed, select 27887 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 27887.
pktcap: Main thread: 287048629056.
pktcap: Dump Thread: 287049164544.
pktcap: Recv Thread: 287049692928.
pktcap: Accept...
pktcap: Vsock connection from port 1028 cid 2.
reading from file -, link-type EN10MB (Ethernet)
05:28:17.947352 [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.947733 [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64
05:28:18.947514 [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.948003 [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
[root@esx-04:~] pktcap-uw --uplink vmnic0 --capture UplinkSndKernel,UplinkRcvKernel -o- | tcpdump-uw -r - -nne | grep -i 10.1.0.10
The name of the uplink is vmnic0.
The session capture point is UplinkSndKernel,UplinkRcvKernel.
pktcap: The output file is -.
pktcap: No server port specifed, select 27896 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 27896.
pktcap: Main thread: 419002739520.
pktcap: Dump Thread: 419003275008.
reading from file -, link-type EN10MB (Ethernet)
pktcap: Recv Thread: 419003803392.
pktcap: Accept...
pktcap: Vsock connection from port 1029 cid 2.
05:28:17.947409 [SRC_TEP_MAC_ADDR] > [DST_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.947716 [SRC_TEP_MAC_ADDR] > [DST_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64
05:28:18.947600 [SRC_TEP_MAC_ADDR] > [DST_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.947968 [SRC_TEP_MAC_ADDR] > [DST_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
[root@esx-05:~] esxcli network vm list | grep -i VM-2
286924 VM-2 1
[root@esx-05:~] esxcli network vm port list -w 286924
Port ID: <port ID>
vSwitch: RegionA01-VDS7
Portgroup:
DVPort ID: <port ID>
MAC Address: MAC_ADDR_2
IP Address: 0.0.0.0
Team Uplink: vmnic0
Uplink Port ID: 2214592517
Active Filters: vmware-sfw
[root@esx-05:~] pktcap-uw --uplink vmnic0 --capture UplinkSndKernel,UplinkRcvKernel -o- | tcpdump-uw -r - -nne | grep -i 10.1.0.10
The name of the uplink is vmnic0.
The session capture point is UplinkSndKernel,UplinkRcvKernel.
pktcap: The output file is -.
pktcap: No server port specifed, select 28035 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 28035.
pktcap: Main thread: 603350494016.
pktcap: Dump Thread: 603351029504.
reading from file -, link-type EN10MB (Ethernet)
pktcap: Recv Thread: 603351557888.
pktcap: Accept...
pktcap: Vsock connection from port <port ID> cid 2.
05:28:17.948445 [SRC_TEP_MAC_ADDR] > [DST_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.948565 [DST_TEP_MAC_ADDR] > [SRC_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64
05:28:18.948662 [SRC_TEP_MAC_ADDR] > [DST_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.11.61837 > 192.168.141.13.6081: Geneve, Flags [C], vni 0x11800, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.948803 [DST_TEP_MAC_ADDR] > [SRC_TEP_MAC_ADDR], ethertype 802.1Q (0x8100), length 160: vlan 141, p 0, ethertype IPv4, 192.168.141.13.50998 > 192.168.141.11.6081: Geneve, Flags [C], vni 0x10802, proto TEB (0x6558), options [8 bytes]: [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
[root@esx-05:~] pktcap-uw --switchport 67108880 --capture VnicTx,VnicRx -o- | tcpdump-uw -r - -nne
The switch port id is 0x04000010.
The session capture point is VnicTx,VnicRx.
pktcap: The output file is -.
pktcap: No server port specifed, select 28025 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 28025.
pktcap: Main thread: 748447206208.
pktcap: Recv Thread: 748448270080.
pktcap: Accept...
pktcap: Vsock connection from port 1030 cid 2.
pktcap: Dump Thread: 748447741696.
reading from file -, link-type EN10MB (Ethernet)
05:28:17.948464 [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1026, length 64
05:28:17.948531 [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1026, length 64
05:28:18.948685 [MAC_ADDR_1] > [MAC_ADDR_2], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 10.1.0.10: ICMP echo request, id 32771, seq 1027, length 64
05:28:18.948769 [MAC_ADDR_2] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 10.1.0.10 > 10.0.0.10: ICMP echo reply, id 32771, seq 1027, length 64
For North-South Traffic
- To capture traffic on ESXi end, the same is applied as stated in "EAST WEST TRAFFIC" section.
- To capture on the Edge Uplink:
-
- Login to the active Edge Node
- get into the vrf of SERVICE_ROUTER_TIER0
vrf <vrf_number> <<<<< entering the respective vrf
get interface <<<<< list all the interfaces and look for "Port-type: uplink" "Interface" "IP" "MAC" to get the Edge uplink interface details
exit
sample:
edge01> get logical-router
Fri Mar 17 2023 UTC 06:21:39.455
Logical Router
UUID VRF LR-ID Name Type Ports Neighbors
736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 4 6/5000
7d50e317-bec3-4d50-91b5-37cb1a28990e 4 16 DR-T1-Gateway-01 DISTRIBUTED_ROUTER_TIER1 8 2/50000
b9cac14c-4c9a-4331-8e0b-40119f821cfb 5 1 DR-T0-Gateway-01 DISTRIBUTED_ROUTER_TIER0 5 2/50000
c4dce003-312e-4c29-8f2a-42326b223dee 6 17 SR-T0-Gateway-01 SERVICE_ROUTER_TIER0 6 2/50000 <<<
edge01> vrf 6
edge01(tier0_sr[6])>
edge01(tier0_sr[6])> get interface
edge01(tier0_sr[6])> get interface
Fri Mar 17 2023 UTC 06:24:10.473
Logical Router
UUID VRF LR-ID Name Type
c4dce003-312e-4c29-8f2a-42326b223dee 6 17 SR-T0-Gateway-01 SERVICE_ROUTER_TIER0
Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable)
Interface : f65978d0-73ba-480b-b984-ec05e4b42a40 <<<<<<
Ifuid : 325
Name : EdgeUplinkB-TN1
Fwd-mode : IPV4_AND_IPV6
Internal name : uplink-325
Mode : lif
Port-type : uplink <<<<<<<<
IP/Mask : 192.168.133.1/24 <<<<<<<<
MAC : ##:##:##:##:##:## <<<<<<<
VLAN : 133
Access-VLAN : untagged
LS port : e87abfd3-c2da-4e20-a106-2babd1d89cd6
Urpf-mode : STRICT_MODE
DAD-mode : LOOSE
RA-mode : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)
Admin : up
Op_state : up
Enable-mcast : False
MTU : 8800
arp_proxy :
edge01(tier0_sr[6])> exit
To see Live Traffic
Sample: pinging from 10.0.0.10 to 8.8.8.8
edge01> start capture interface f65978d0-73ba-480b-b984-ec05e4b42a40 direction dual expression host 10.0.0.10
06:30:37.592896 [MAC_ADDR_1] > [DEST_MAC], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 8.8.8.8: ICMP echo request, id 32779, seq 1030, length 64
<base64>AFBWAQb4AFBWnZzFCABFwAA0AAAAAP8RLqjAqIUBwKiF/tv0DsgAIK3iIMADGCF4DxuO43LQAA9CQAAPQkAAAAAA</base64>
06:30:38.433363 [DEST_MAC] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 10.0.0.10: ICMP echo reply, id 32779, seq 1030, length 64
<base64>AFBWAQb4AFBWnZzFCABFwAA0AAAAAP8RLqjAqIUBwKiF/tv0DsgAIK3iIMADGCF4DxuO43LQAA9CQAAPQkAAAAAA</base64>
06:30:38.743828 [MAC_ADDR_1] > [DEST_MAC], ethertype IPv4 (0x0800), length 98: 10.0.0.10 > 8.8.8.8: ICMP echo request, id 32779, seq 1031, length 64
<base64>AFBWAQb4AFBWnZzFCABFwAA0AAAAAP8RLqjAqIUBwKiF/tv0DsgAIK3iIMADGCF4DxuO43LQAA9CQAAPQkAAAAAA</base64>
<base64>AFBWnZzFAFBWAQb4CABFwAA0JBFAAP8RypbAqIX+wKiFAcAcDsgAIAZqIMADGI7jctAheA8bAA9CQAAPQkAAAMNQ</base64>
06:30:39.382513 [DEST_MAC] > [MAC_ADDR_1], ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 10.0.0.10: ICMP echo reply, id 32779, seq 1031, length 64
<base64>AFBWAQb4AFBWnZzFCABFwAA0AAAAAP8RLqjAqIUBwKiF/tv0DsgAIK3iIMADGCF4DxuO43LQAA9CQAAPQkAAAAAA</base64>
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
To Capture and Write on pcap
- Define capture session
set capture session <session-number> interface <port-uuid-name> direction dual
- View capture session
get capture session
- Start capture session
set capture session <session-number> file <filename> expression <expression>
- View capture files
get files
- Copy capture files
copy file <filename> url scp://username@ip_address/filepath/filename
Alternatively, you can just scp using any client (WinSCP) etc to the NSX-T edge node and extract the files from stored directory.
Note: generated pcaps are stored in "/image/vmware/nsx/file-store/" directory on NSX-T edge node.
Sample
edge01> set capture session 0 interface f65978d0-73ba-480b-b984-ec05e4b42a40 direction dual
edge01> get capture session
Fri Mar 17 2023 UTC 06:40:03.244
Packet Capture Session
ID : 0
PORTS : ['<ports>']
Packet Capture Session
ID : 1
PORTS : []
Packet Capture Session
ID : 2
PORTS : []
Packet Capture Session
ID : 3
PORTS : []
Packet Capture Session
ID : 4
PORTS : []
Packet Capture Session
ID : 5
PORTS : []
edge01> set capture session 0 file Test_Capture.pcap expression host 192.168.133.254
Capture to file initiated, enter Ctrl-C to terminate
^C11 packets captured
12 packets received by filter
0 packets dropped by kernel
edge01> get files
Fri Mar 17 2023 UTC 06:43:04.413
Directory of filestore:/
-rw- 5398 Nov 03 2022 23:52:31 UTC nsx_backup_cleaner.py
-rw- 9967 Nov 03 2022 23:52:31 UTC backup_restore_helper.py
-rw- 972 Mar 17 2023 06:42:42 UTC Test_Capture.pcap <<<<<<<<<<
-rw- 5748 Nov 03 2022 23:52:31 UTC get_backup_timestamps.sh
edge01> copy file Test_Capture.pcap url scp://[email protected]/tmp/
Are you sure you want to continue connecting (yes/no)? yes
Password:
Test_Capture.pcap 100% 972 1.2MB/s 00:00
edge01>
Feedback
