Network traffic loss for VMs on a specific port group when port mirroring is enabled.

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

Packet loss is observed for VMs on a specific port group or VLAN.

ARP requests from an affected VM do not leave the host.

You are using VMware VDS and have port mirroring enabled in your environment.

Environment

VMware vSphere ESXi

Cause

Perform the below steps to trace packets for this VM. 

1. Find the switchport-id for the VM using the command: net-stats -l | grep -i <vm_name>
2. Start a continuous ping on VM to VM gateway.
3. Perform a packet capture on ESXi host running this VM.
   This is showing ARP packets coming from the VM and entering the vDS.
   

   • pktcap-uw --switchport <vm_switchport-id> --capture PortInput,PortOutput -o - | tcpdump-uw -r - -ean

 

Ctrl-C to end the trace.

     4. Now check if packets are leaving the vDS.

• • Run: netdbg vswitch instance list | grep -i <vm_name> 
  • Take note of the vmnic the VM is attached to.
  • Run the below packet capture.
    pktcap-uw --uplink <vmnic#> --capture PortInput,PortOutput -o - | tcpdump-uw -enr - host <ip of VM>
  • This will show no packets leaving the vDS.

Reviewing vmkernel logs on the host (/var/run/log/vmkernel.log) shows a port mirror session being added to vDS around the same time the port is enabled for the VM with mac 00:50:56:##:##:##.

2024-11-11T18:32:46.798Z In(182) vmkernel: cpu33:5140587)NetPort: 1610: enabled port 0x600005d with mac 00:50:56:##:##:##

2024-11-11T18:32:46.797Z In(182) vmkernel: cpu33:5140587)Mirror.cswitch: VSwitchMirrorSessionAdd:4585: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session <port_mirror_name>: succeeded to add session

2024-11-11T18:32:46.797Z In(182) vmkernel: cpu33:5140587)Mirror.cswitch: Mirror_UpdateSessionsFromDVPortData:8330: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session <port_mirror_name>: succeeded to add light session

 

Using vCenter GUI, navigate to the "ports" tab of the portgroup in use. Capture the port ID the VM is attached to.

\

Port 10 is where the VM is attached to vDS.

Navigating back to ESXi SSH session and run the below command using the port ID number found above.

net-dvs -l | grep "port <portnumber>:" -A 100 | grep -i mirror -A 10

This will show that a port mirror config is setup on this port:

example output:
com.vmware.vswitch.port.mirrorSessions:
                        sid:                    0
                        dname:                  <port_mirror_name>
                        flags:                  DST_NO_OUTPUT DST_NO_INPUT DST_ORIG_VLAN
                        snapLen:                -1 (full frame)
                        encapVlan:              0 (no ENCAP)
                        type:                   DVPORT
                        samplingRate:           2

                        dstEntities     (1):    10
                        srcInEntities   (2):    98 4

 

Resolution

Due to port mirroring being set to disallow I/O, the packets are discarded in the vDS. 

To resolve this issue you may:

1. Remove VM port ID from being used in this mirror
2. Delete the port mirror
3. Set the "Normal I/O on destination ports" flag to "Allowed" in port mirror config in VC UI.

Additional Information

If the above scenario does not fit your particular situation, please peruse KB: Troubleshooting virtual machine network connection issues

Create Port Mirroring Session