High packet rate and network bandwidth utilization is observed after configuring port mirroring for a vmkernel portgroup using "Encapsulated Remote Mirroring (L3) Source" session type

Products

• VMware

Issue/Introduction

Symptoms:

• The dvPort that a vmkernel interface is connected to is configured as the source port for the port mirroring session AND the port mirroring session type is configured as "Encapsulated Remote Mirroring (L3) Source"
• You see an excessive amount of network traffic being generated by a vmkernel interface (usually vmk0) after enabling port mirroring with session type "Encapsulated Remote Mirroring (L3) Source"
• After enabling port mirroring on the DVS using session type "Encapsulated Remote Mirroring (L3) Source", you notice a significant traffic increase on the physical network

Cause

When the port mirroring session type Encapsulated Remote Mirroring (L3) Source is used, a vmkernel interface is used to encapsulate traffic to the remote mirroring destination specified.  The vmkernel used to send this traffic will depend on the remote IP address specified and the vmkernel routing table.  When the dvPort being mirrored contains the vmkernel interface that is used to encapsulate the mirrored traffic, the normal traffic on the interface is mirrored as well as the port mirroring traffic.  This compounds as mirrored packets are mirrored again and again, leading to the bandwidth saturation of the physical uplink and a high packet rate being generated by this vmkernel interface.

Resolution

Avoid mirroring traffic for vmkernel interfaces using a port mirroring session with the "Encapsulated Remote Mirroring (L3) Source" session type.

If using vSphere 6.7, you can work around this issue by configuring a separate vmkernel TCP/IP stack for port mirroring and reconfigure the DVS port mirroring session to use this TCP/IP stack.  To perform this workaround:

1. From an SSH session to the ESXi host, add the port mirroring TCP/IP stack with the following command: esxcli network ip netstack add -N mirror
2. Using vSphere Web Client, navigate to Home -> Inventory -> Hosts and Clusters, select the ESXi host in inventory and navigate to Configure -> VMkernel adapters.  Add a new vmkernel adapter and in the Port properties part of the wizard be sure to change the TCP/IP stack to mirror from the dropdown menu.
3. Using the HTML5 vSphere Web Client client, navigate to Home -> Inventory -> Networking, select the distributed switch in inventory and navigate to Configure -> Port Mirroring.  Edit the Properties of the port mirroring session that contains the vmkernel interface and change the TCP/IP stack to Mirror using the drop-down menu.

Additional Information

To check what vmkernel interface is used to forward traffic to a given IP address, check the ESXi host routing table:

[root@esxi-1:~] esxcli network ip route ipv4 list
Network       Netmask        Gateway      Interface  Source
------------  -------------  -----------  ---------  ------
default       0.0.0.0        192.168.0.1  vmk0       MANUAL
192.168.0.0   255.255.255.0  0.0.0.0      vmk0       MANUAL
192.168.10.0  255.255.255.0  0.0.0.0      vmk1       MANUAL
192.168.20.0  255.255.255.0  0.0.0.0      vmk2       MANUAL

Then compare the IP address configured as the remote mirroring destination.  If the destination IP address matches one of the networks in the routing table, this interface will be used to send the traffic.  If the IP address does not match an entry in the routing table, the vmkernel interface with the default route configured will be used.