Intermittent ICMP Packet Drops from a VM to its gateway and also to its own adaptor IP

Products

• VMware vCenter Server

Issue/Introduction

- Virtual machines experience intermittent ICMP packet drops when pinging their own local network interfaces and their default gateway.

- There is a Palo Alto Cortex XDR end point protection software

- Packet captures for the ICMP at Vnic of the VM shows that there are only few ICMP requests coming from the VM based on sequence number of the ICMP requests indicating the packets are getting drops are with in the VM itself

As an example, trace a particular vSwitch port:

1. To get the vSwitch port number of this VM:
   
   net-stats -l | grep <vm-name>

2. Identify and make a note of these parameters:

   Port ID returned by the esxtop command — --switchport <switchport-id>

   Location of the output pcap file — /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_12345678/

3. Run the pktcap-uw command to capture packets, ensure to replace the ####### with the specific switchport that the capture is going to be taken on.:
   
   pktcap-uw --switchport ######## -o /vmfs/volumes/FULL_PATH_TO_DATASTORE/Case_#/esxi01.switchport.########.pcap

Environment

VMware vSphere ESXi

Cause

Conflict between VMware Tools and Cortex Endpoint Protection (XDR) software on a Virtual Machine (VM) can cause network interface drops, especially during high activity or when VMware tools attempt to update network drivers. This issue is often characterized by the Cortex XDR agent falsely flagging VMware’s vmtoolsd.exe or the network driver itself as a threat, leading to session termination

Resolution

Steps to resolve:

• Upgrade VMware Tools on the affected virtual machine to the latest version compatible with the ESXi host.

• Engaging the end point security vendor to make sure the agent is also updated to the latest versions.

• Disable and re-enable the network interface within the guest operating system to reset the connection state.