Using pktcap-uw to troubleshoot ping issues to a virtual machine specific distributed virtual switch.

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Unable to ping vm on non-nsx backed networks.
• Other virtual machines (VM) on same Distributed Virtual Switch (DVS) network cannot ping some virtual machines (VM) if the virtual machine is on a different hosts.
• Firewall is allowing ICMP traffic.
• VM is Pingable on same host as another VM.

Environment

7.x

8.x

Resolution

Identifying the active uplink carrying a specific vm/vmk traffic:

1. Run the command "net-stats -l" and this will list the switchports that are mapped to vm/vmk.
   
   
2. Option 1 to identify the uplink: run the command "esxtop" and press "n". This will show the mapping between switchport and uplink.
3. Option 2 to identify the uplink: run the command "vsish" and press enter. "cat /net/portsets/vSwitch_name/ports/switchport_number/teamUplink"
   Note: For vSwitch name and switchport number, refer to "net-stats -l" output.
4. Option 3 to identify the uplink: run the command "netdbg vswitch instance list" and press enter.
   
   

Run pktcap-uw to isolate traffic flow:

1. 1. SSH to ESXi host where Destination VM (via switchport id) is being pinged and Run:
      pktcap-uw –switchport <switchport> --dir 2 -o - | tcpdump-uw -enr -| grep -i icmp
      1. If you get a response showing similar to:
         <Timestamp><Source-Mac> > <Destination-Mac> ethertype <type>, Length #: <Source-IP> > <Destination-IP> <Protocol> echo <request or reply>, <id>, seq #, length #
         
         Traffic will make it to nic and vm but not to the guest.
         
         
      2. If you get no response output after running the command, then the traffic is not making it to the physical nic
         
         1. You will want to additionally check the vmnic# (physical nic of host) via:
            esxcli network nic stats get -n vmnic#
            This will ensure you have no issues with physical nic itself
            Example of healthy output:
               Packets received: 701280499176
   Packets sent: 687061948450
   Bytes received: 664124780523852
   Bytes sent: 676938646792793
   Receive packets dropped: 2452783244
   Transmit packets dropped: 0
   Multicast packets received: 976222150
   Broadcast packets received: 0
   Multicast packets sent: 0
   Broadcast packets sent: 0
   Total receive errors: 0
   Receive length errors: 0
   Receive over errors: 0
   Receive CRC errors: 0
   Receive frame errors: 0
   Receive FIFO errors: 0
   Receive missed errors: 0
   Total transmit errors: 0
   Transmit aborted errors: 0
   Transmit carrier errors: 0
   Transmit FIFO errors: 0
   Transmit heartbeat errors: 0
   Transmit window errors: 0
         2. Once you have ruled out that the nic itself is getting traffic and no issues present themselves this issue becomes an upstream investigation issue (Top Of Rack Switch, Router, etc.)

Additional Information

For More information on commands from above see:
Packet capture on ESXi using pktcap-uw
Troubleshooting NIC errors and other network traffic faults in ESXi