Packet loss is observed when pinging the Default Gateway from an ESXi host

Products

• VMware vSphere ESXi

Issue/Introduction

• When a continuous ping is initiated to the default gateway from the ESXi host, few percentage of packet loss is observed as follows:

• No Packet drops/errors are seen on the Physical Network Adapter used by VMkernel adapter where Management Service is enabled.
  • Command to check the stats: esxcli network nic stats get -n vmnicX

NIC statistics for vmnicX
   Packets received: 49198985
   Packets sent: 49786477
   Bytes received: 8770188340
   Bytes sent: 12513743409
   Receive packets dropped: 0
   Transmit packets dropped: 0
   Multicast packets received: 25300171
   Broadcast packets received: 3391039
   Multicast packets sent: 15526
   Broadcast packets sent: 695
   Total receive errors: 0
   Receive length errors: 0
   Receive over errors: 0
   Receive CRC errors: 0
   Receive frame errors: 0
   Receive FIFO errors: 0
   Receive missed errors: 0
   Total transmit errors: 0
   Transmit aborted errors: 0
   Transmit carrier errors: 0
   Transmit FIFO errors: 0
   Transmit heartbeat errors: 0
   Transmit window errors: 0

Environment

VMware vSphere ESXi

Resolution

To isolate the packet loss, perform the following steps:

• Open Out-of-Band Management access [iLO/iDRAC] of the impacted ESXi host.
• Press F2 key to login.
• Go to Troubleshooting Options.
• Press Enter on Enable ESXi Shell.
• Press Alt+F1 key combination to access the ESXi Shell.
• Login using root credentials.
• Find out the Physical Network Adapter used by the Management VMkernel adapter using the following command [This Network adapter will be used later in the packet capture command]: 

netdbg vswitch instance list

• Output will be as follows:

[root@esx-01:~] netdbg vswitch instance list | grep vmk0
  vmk0                           <switchport-id>        <dvPort-id>                                   00:50:56:##:##:##    vmnic0          <vlan-id>              N/A

• Initiate a ping to the gateway along with packet captures using the following command:

ping <gateway-ip-address> -c 1000 & pktcap-uw --uplink vmnic0 --dir 2 -o /vmfs/volumes/<datastore-name>/uplink.pcapng

• Once the pings are finished, press Ctrl+C to stop the captures.
• Download the capture file uplink.pcapng

To download the file, navigate to the datastore on the VCenter UI. Click on the files tab. Fetch the file and click on Download.

Once the packet capture file is downloaded, analyze the ICMP stream (using a tool like Wireshark) to determine packet drops:

Scenario A: ICMP Request Sent, No ICMP Reply received (Physical Network Issue)

• Observation: You see the ICMP Echo Requests leaving the ESXi host (Outbound), but there are no corresponding Echo Replies coming back (Inbound) for the dropped sequence numbers.

• Action: This indicates the ICMP Request packet successfully left the ESXi host but either the ICMP Request was dropped before it reached the default gateway or ICMP Reply was dropped before it reached the ESXi host. Engage your Physical Network Team to investigate the upstream switch, routing, or firewall configuration.

Scenario B: ICMP Request Missing (ESXi host issue)

• Observation: You see a gap in the ICMP Echo Request sequence numbers (e.g., Sequence 10, 11, [Missing 12], 13) within the capture itself, meaning the packet never reached the ESXi host's uplink.

• Action: Open a Support Case with Broadcom, referencing this KB article. Please upload the uplink.pcapng file and the ESXi Support Bundle to the case.

 

NOTE: If you need assistance in analyzing the packet captures, you can reach out to Broadcom Support or engage your Networking team.