Multicast flows stopped working on ENS enabled NSX transport nodes

Products

• VMware NSX

Issue/Introduction

Issue Symptoms

• Traffic using multicast flows got impacted when the limit of 8 destination multicast ports is reached on an ENS enabled cluster
• This issue was seen with OSPF traffic as all the OSPF neighbors went down with Edge nodes when the limit of 8 destination ports is reached 
• All OSPF multicast flows will be impacted when more than 8 destination ports (4 edges ( 2 uplink)/ 8 edges (single uplinks) are added on the Host with ENS enabled.

Environment

NSX-T 3.2.2
ESXi 7.0.3 23307199

Cause

When latency metric collection was enabled on vRNI, traceflow was enabled on ESX host where Edge nodes were deployed to enable datapath metrics collections from edge nodes. As traceflow was enabled, all multicast packets (OSPF control) were directed to flow on non-ENS path. 

When a new service config (to disable latency metric collection) was created and applied from vRNI, a latency profile with traceflow disabled got applied on ESX hosts where Edge nodes were hosted. As a result, ENS started to process OSPF packets in fast path. By design and implementation, ENS has a limitation of having only 8 destination ports for each multicast flow resulting in OSPF down for the Edge nodes with its uplink neighbors.

In general this issue could be seen with all multicast traffic where the multicast flows get impacted when the limit of 8 destination ports is reached. 

Resolution

This issue is resolved in NSX 3.2.5 , 4.1.1 and 4.2.X

 

Additional Information

Initial checks on ESX host 

• To check if the Transport Node has ENS supported NICs and drivers, login to the ESXi host and execute below command 

> esxcfg-nics -e 

From the output, if the column "ENS capable" & "ENS Driven" is set to "True" for the physical vmnicsX then that ESXI host is enabled with ENS  

• To check if ENS multicast flow is enabled in datapath, execute below command on ESXi host

> net-dvs -l | grep -i fc.mcast

From the output check if "com.vmware.net.portset.fc.mcast.enabled is set to "true" then multicast flow in enabled in datapath 

• To check if cfgagent has the default setting of "True" set for multicast in flowcache 

> cat /etc/vmware/nsx/nsx-cfgAgent.xml 

From the output check if "mcastEnabled" is set to "True" as shown below 

             <features>                                                          
                <flowCache>                                                      
                   <enabled>true</enabled>                                       
                   <mcastEnabled>true</mcastEnabled>  
                </flowCache>                                                     
             </features>

• To check if there are multicast based flows in this dump marked as invalid flows 

> nsxdp-cli ens flow-table dump

Flows marked with MC will show that Multicast is enabled and in issue state, all of multicast flows are marked as invalidated.