Virtual machines sporadically losing network connectivity

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machines (VMs) intermittently lose layer 2 connectivity and cannot communicate on the network.
• Issue affects VMs on the same host and same portgroup.
• Migrating VMs to other hosts does not resolve the connectivity loss and modifying the portgroup doesn't have any impact on VM connectivity.
• Port Mirroring is enabled on the virtual switch connected to affected VMs.
• The ESXi host /var/log/vmkernel.log report details for port mirroring sessions created with flags: DST_NO_OUTPUT DST_NO_INPUT DST_ORIG_VLAN.

<TIMESTAMP> In(182) vmkernel: cpu89:17316280)Mirror.cswitch: VSwitchMirrorSessionAdd:4743: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session <PORT_MIRROR_NAME>-port-mirroring__host-<ID>: succeeded to add session
<TIMESTAMP> In(182) vmkernel: cpu89:17316280)Mirror.cswitch: Mirror_UpdateSessionsFromDVPortData:8510: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session Gigamon-port-mirroring__host-<ID>: succeeded to add light session
<TIMESTAMP> In(182) vmkernel: cpu89:17316280)NetPort: <ID>: enabled port 0x60001b5 with mac <MAC_ADDRESS>
port <ID>:

com.vmware.common.port.alias =  ,       propType = CONFIG
                com.vmware.common.port.connectid = <ID> ,         propType = CONFIG
                com.vmware.common.port.volatile.status = inUse linkUp portID=<ID>  propType = RUNTIME
                com.vmware.common.port.volatile.vlan = VLAN <ID>
                        propType = RUNTIME
                com.vmware.common.port.portgroupid = dvportgroup-<ID>,         propType = CONFIG
                com.vmware.common.port.block = false ,  propType = CONFIG
                com.vmware.common.port.dvfilter = filters (num = 0):
                        propType = CONFIG
                com.vmware.vswitch.port.teaming.policy:
                        .....
                        .....
                com.vmware.common.port.volatile.persist = /vmfs/volumes/vsan:<ID>/.dvsData/<ID>/<ID> , propType = CONFIG
                com.vmware.vswitch.port.mirrorSessions:
                        sid: 23
                        dname: Gigamon-port-mirroring__host-<ID>
                        flags: DST_NO_OUTPUT DST_NO_INPUT DST_ORIG_VLAN
                        snapLen: -1 (full frame)
                        encapVlan: 0 (no ENCAP)
                        type: DVPORT
                        samplingRate: 1
                        dstEntities (1): 2858
                        srcInEntities (1): 588
                        srcOutEntities (1): 588
                        propType = CONFIG
                com.vmware.common.port.volatile.cleanupBarrier = 1764900071 , propType = VOLATILE
                com.vmware.common.port.volatile.ptstatus = noPassthruReason=32, propType = RUNTIME
                com.vmware.vswitch.port.macLearningStats = 0x 0. 0 <repeats 27 times>
                        propType = RUNTIME

Cause

The configured port mirroring was dropping all traffic for the VMs over the configured port.

Resolution

To resolve the connectivity issue, update / remove the port mirroring session in vCenter so that it doesn't drop network traffic. 

To view port mirroring sessions configured on a distributed switch, in the vSphere Client go to Inventory > Networks tab > Distributed Switch > Configure > Port Mirroring.

Additional Information

For more information regarding the configuration of port mirroring session, please refer: Create a Port Mirroring Session