Virtual machines sporadically losing network connectivity

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machines (VMs) intermittently lose layer 2 connectivity and are unable to communicate on the network
• Affects VMs on the same host and the same portgroup
• Migrating the VMs to other hosts does not resolve the issue
• Modifying the VMs' portgroup settings has no effect
• Port Mirroring is enabled on the virtual switch that the affected VMs are connected to
• On the ESXi host's /var/log/vmkernel.log file you see a port mirroring session being created with the flags DST_NO_OUTPUT DST_NO_INPUT DST_ORIG_VLAN set:

  <TIMESTAMP> In(182) vmkernel: cpu89:17316280)Mirror.cswitch: VSwitchMirrorSessionAdd:4743: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session <PORT_MIRROR_NAME>-port-mirroring__host-<ID>: succeeded to add session
<TIMESTAMP> In(182) vmkernel: cpu89:17316280)Mirror.cswitch: Mirror_UpdateSessionsFromDVPortData:8510: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]ps DvsPortset-1, session Gigamon-port-mirroring__host-<ID>: succeeded to add light session
<TIMESTAMP> In(182) vmkernel: cpu89:17316280)NetPort: <ID>: enabled port 0x60001b5 with mac <MAC_ADDRESS>
port <ID>:


  com.vmware.common.port.alias =  ,       propType = CONFIG
                  com.vmware.common.port.connectid = <ID> ,         propType = CONFIG
                  com.vmware.common.port.volatile.status = inUse linkUp portID=<ID>  propType = RUNTIME
                  com.vmware.common.port.volatile.vlan = VLAN <ID>
                          propType = RUNTIME
                  com.vmware.common.port.portgroupid = dvportgroup-<ID>,         propType = CONFIG
                  com.vmware.common.port.block = false ,  propType = CONFIG
                  com.vmware.common.port.dvfilter = filters (num = 0):
                          propType = CONFIG
                  com.vmware.vswitch.port.teaming.policy:
                          load balancing = source port traffic load
                          link selection = link state up;
                          link behavior = notify switch; best effort on failure; shotgun on failure;
                          active = Uplink 1, Uplink 2,
                          standby =
                          propType = CONFIG
                  com.vmware.vswitch.port.security = deny promiscuous; deny mac change; deny forged frames
                          propType = CONFIG
                  com.vmware.vswitch.port.vlan = VLAN <ID>
                          propType = CONFIG
                  com.vmware.vswitch.port.txUplink = normal ,     propType = CONFIG
                  com.vmware.common.port.statistics:
                          pktsInUnicast = 31203520665
                          bytesInUnicast = 66884783499156
                          pktsInMulticast = 164597
                          bytesInMulticast = 15195219
                          pktsInBroadcast = 391772
                          bytesInBroadcast = 25179926
                          pktsOutUnicast = 131279481307
                          bytesOutUnicast = 114494148309366
                          pktsOutMulticast = 632663
                          bytesOutMulticast = 48865076
                          pktsOutBroadcast = 146643
                  com.vmware.common.port.volatile.persist = /vmfs/volumes/vsan:<ID>/.dvsData/<ID>/<ID> , propType = CONFIG
                  com.vmware.vswitch.port.mirrorSessions:
                          sid: 23
                          dname: Gigamon-port-mirroring__host-<ID>
                          flags: DST_NO_OUTPUT DST_NO_INPUT DST_ORIG_VLAN
                          snapLen: -1 (full frame)
                          encapVlan: 0 (no ENCAP)
                          type: DVPORT
                          samplingRate: 1
                          dstEntities (1): 2858
                          srcInEntities (1): 588
                          srcOutEntities (1): 588
                          propType = CONFIG
                  com.vmware.common.port.volatile.cleanupBarrier = 1764900071 , propType = VOLATILE
                  com.vmware.common.port.volatile.ptstatus = noPassthruReason=32, propType = RUNTIME
                  com.vmware.vswitch.port.macLearningStats = 0x 0. 0 <repeats 27 times>
                          propType = RUNTIME

Cause

The port mirroring configuration is dropping all traffic.

Resolution

Update / remove the port mirroring session in vCenter so that it doesn't drop network traffic. 

You can view port mirroring sessions for a distributed switch in the vSphere Client by going to Inventory > Networks tab > Distributed Switch > Configure > Port Mirroring.