Some ENS flows are incorrectly appearing under the wrong flow category

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

For example, we may see ENS flow cache using L4 flows instead of L2 flows but only in the Rx direction (from VM's point of view). Host is configured with overlayL2Flow so all flows should be L2.
You can verify this specific parameter with the following command 
  nsxdp-cli ens switch overlayL2Flow get -dvs <ENS-DVS-NAME>


To check the various categorized flow you need to use those commands 
nsxdp-cli ens flow-table dump -l <logical core number>

You can grep the specific type of flow  (L2/L3/L4) 
nsxdp-cli ens flow-table dump -l <logical core number> | grep L2


such flow distribution could quickly overrun the flow cache leading to performance problems on the ENS side  , depending on the application traffic.

Environment

NSX used on ESXi  (no specific version)

Cause

The reason for this behaviour  is LACP is configured such that it requires  different ENS flows (e.g. ENS L4 flows instead of L2 in the example below). In this specific case, if ENS L2 were desired a MAC/VLAN load balancing algorithm should have been chosen 

Using the command 
 esxcli network vswitch dvs vmware lacp config get     ​​ 

DVS Name                  LAG Name   LAG ID       NICs           Enabled  Mode         Load balance
------------------------  ---------- --------    -------------   -------  -------      -------------------------- 
VDS-RESOURCE              lag1       1234567890   vmnic2,vmnic3  true     Passive      Src and dst ip, port, vlan
VDS-MGT                   lag2       9087654321   vmnic0,vmnic1  true     Passive      Src and dst ip, port, vlan


We can see the current LACP load balancing configuration for DVSs having LACP configured.

Resolution

Configure the appropriate load balancing for the desired flow categorization according to this  this table

ENS_FLOW_L2 LACP LB modes
LB_NONE
LB_DSTMAC
LB_SRCMAC
LB_SRCMAC_DSTMAC
LB_VLAN
LB_SRCID

ENS_FLOW_L3 LACP LB modes
LB_SRCIP
LB_DSTIP_VLAN
LB_SRCIP_DSTIP
LB_SRCIP_VLAN
LB_SRCIP_DSTIP_VLAN

ENS_FLOW_L4  LACP LB modes
LB_DSTPORT
LB_SRCPORT
LB_SRCPORT_DSTPORT
LB_DSTIP_DSTPORT
LB_SRCIP_SRCPORT
LB_SRCIP_DSTIP_SRCPORT_DSTPORT
LB_DSTIP_DSTPORT_VLAN
LB_SRCIP_SRCPORT_VLAN
LB_SRCIP_DSTIP_SRCPORT_DSTPORT_VLAN