VM network traffic drops when on a specific port group

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

When VM is running on problematic portgroup/vDS you will observe via packet-traces that packets for this VM do not leave the vDS.

ARP requests from an affected VM do not leave the host.

Cause

Perform the below steps to trace packets for this VM.

Find the switchport-id for the VM using the command: net-stats -l | grep -i <vm_name>
Start a continuous ping on VM to VM gateway.
Perform a packet capture on ESXi host running this VM.
This is showing ARP packets coming from the VM and entering the vDS.
- ```
pktcap-uw --switchport <vm_switchport-id> --capture PortInput,PortOutput -o - | tcpdump-uw -r - -ean
```

Ctrl-C to end the trace.

4. Now check if packets are leaving the vDS.

- Run: netdbg vswitch instance list | grep -i <vm_name>
- Take note of the vmnic the VM is attached to.
- Run the below packet capture.
  Note: host x.x.x.x = VM IP
```
pktcap-uw --uplink <vmnic_#> --capture PortInput,PortOutput -o - | tcpdump-uw -r - -ean host x.x.x.x
```
- This will show no packets leaving the vDS.

Reviewing vmkernel logs on the host (/var/run/log/vmkernel.log) shows a port mirror session being added to vDS around the same time the port is enabled for the VM with mac 00:50:56:##:##:##.

2024-11-11T18:32:46.798Z In(182) vmkernel: cpu33:5140587)NetPort: 1610: enabled port 0x600005d with mac 00:50:56:##:##:##

2024-11-11T18:32:46.797Z In(182) vmkernel: cpu33:5140587)Mirror.cswitch: VSwitchMirrorSessionAdd:4585: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session <port_mirror_name>: succeeded to add session

2024-11-11T18:32:46.797Z In(182) vmkernel: cpu33:5140587)Mirror.cswitch: Mirror_UpdateSessionsFromDVPortData:8330: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session <port_mirror_name>: succeeded to add light session

Using vCenter GUI, navigate to the "ports" tab of the portgroup in use. Capture the port ID the VM is attached to.

\

Port 10 is where the VM is attached to vDS.

Navigating back to ESXi SSH session and run the below command using the port ID number found above.

net-dvs -l | grep "port 10:" -A 100 | grep -i mirror -A 10

This will show that a port mirror config is setup on this port:

com.vmware.vswitch.port.mirrorSessions:
                        sid:                    0
                        dname:                  <port_mirror_name>
                        flags:                  DST_NO_OUTPUT DST_NO_INPUT DST_ORIG_VLAN
                        snapLen:                -1 (full frame)
                        encapVlan:              0 (no ENCAP)
                        type:                   DVPORT
                        samplingRate:           2

                        dstEntities     (1):    10
                        srcInEntities   (2):    98 4

Resolution

Due to port mirroring being set to disallow I/O, the packets are discarded in the vDS.

To resolve this issue you may:

Remove VM port ID from being used in this mirror
Delete the port mirror
Set the "Normal I/O on destination ports" flag to "Allowed" in port mirror config in VC UI.

Additional Information

If the above scenario does not fit your particular situation, please peruse KB: Troubleshooting virtual machine network connection issues

Create Port Mirroring Session