NSX-T Edge VM on NSX-V prepared host may experience Datapath Impact if not added to DFW Exclusion List in NSX-V

Products

VMware NSX

Issue/Introduction

In instances where an NSX-T Edge VM is deployed on an NSX-V prepared host, traffic flowing through this Edge VM might potentially hit a Distributed Firewall (DFW) deny rule in NSX-V and must be added to the DFW Exclusion List.

Note: It is common to have an NSX-T Edge VM deployed as a Bridge on an NSX-V prepped host when performing a V2T migration. Please see the following documentation for more details.

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/migration/GUID-962933BC-2E55-411E-8094-C19CD15983A4.html

Symptoms:

Workloads in an NSX-T environment are experiencing impact to traffic for Northbound/Southbound traffic. Traffic Northbound is leaving the NSX-T edge uplink but not seen on the vmnic of the ESXi host where the Edge VM is deployed.

When logging in to the ESXi host via SSH and reviewing the installed VIBs, it is found that the esx-nsxv VIB is installed.

esxcli software vib list | grep -i 'nsx'
esx-nsxv                       6.7.0-0.0.18516827                    VMware   VMwareCertified   2021-10-08

If the command output shows nsx-v vibs the host where the NSX-T Edge VM is deployed (as above), then proceed to check the dvfilter for the Edge VM and if there are any attached rules. Below we see that there is a filter attached at slot 2 for each of the interfaces on the manually deployed NSX-T Edge OVA/OVF.

[root@ds-tse-d44:~] summarize-dvfilter | grep -iA24 demo-nsxt-edge
world 4940159 vmm0:demo-nsxt-edge-3.2.1.0.0.19801966 vcUuid:'<UUID>'
 port 100663330 demo-nsxt-edge-3.2.1.0.0.19801966.eth4
  vNic slot 2
   name: nic-4940159-eth4-vmware-sfw.2        <---- Filter name
   agentName: vmware-sfw
   state: IOChain Attached
   vmState: Attached                          <---- filter is attached, need to check rules 
   failurePolicy: failClosed
   serviceVMID: 2
   filter source: Dynamic Filter Creation
  vNic slot 1
   name: nic-4940159-eth4-dvfilter-generic-vmware-swsec.1
   agentName: dvfilter-generic-vmware-swsec   <---- this is a different filter for L2 security (spoofguard). This is outside the scope of this KB and is not of concern.
   state: IOChain Attached
   vmState: Detached
   failurePolicy: failClosed
   serviceVMID: none
   filter source: Alternate Opaque Channel

After adding the Edge VM to the NSX-V Exclusion list, the state that we want to see the Edge VM in is as follows. This indicates that the L3 DFW rules have been removed and the dvfilters at slot 2 on the vnic have been removed.

[root@ds-tse-d44:~] summarize-dvfilter | grep -iA32 demo-nsxt-edge
world 4940159 vmm0:demo-nsxt-edge-3.2.1.0.0.19801966 vcUuid:'<UUID>'
 port 100663331 demo-nsxt-edge-3.2.1.0.0.19801966.eth3
  vNic slot 1                                <---- Only slot 1 is attached now. Slot 2 is detached.
   name: nic-4940159-eth3-dvfilter-generic-vmware-swsec.1
   agentName: dvfilter-generic-vmware-swsec
   state: IOChain Attached
   vmState: Detached
   failurePolicy: failClosed
   serviceVMID: none
   filter source: Alternate Opaque Channel
 port 100663330 demo-nsxt-edge-3.2.1.0.0.19801966.eth4
  vNic slot 1
   name: nic-4940159-eth4-dvfilter-generic-vmware-swsec.1
   agentName: dvfilter-generic-vmware-swsec
   state: IOChain Attached
   vmState: Detached
   failurePolicy: failClosed
   serviceVMID: none
   filter source: Alternate Opaque Channel

Environment

VMware NSX-T Data Center

Cause

The Distributed Firewall (DFW) doesn't differentiate VMs unless it is told to do so. For a typical NSX manager deployed VM, the VM being deployed is typically automatically added to the Firewall Exclusion list. However, for instances where you have to manually deploy an NSX-T VM (NSX Manager appliance, Edge VM, etc.), such as setting up an Edge bridge for V2T migrations, those VMs are not automatically added to the Exclusion list.

Resolution

To avoid this issue, if performing a V2T migration where an NSX-T Edge Bridge is deployed on an NSX-V prepared host, the NSX-T Edge VM facilitating the bridge must be added to the NSX-V Distributed Firewall (DFW) Exclusion List.

NSX-V DFW Exclusion list Documentation:

https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/com.vmware.nsx.install.doc/GUID-C3DDFBCE-A51A-40B2-BFE1-E549F2B770F7.html

NSX-T DFW Exclusion list Documentation:

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/migration/GUID-962933BC-2E55-411E-8094-C19CD15983A4.html

Note: It is recommended to include the Edge VM in the DFW Exclusion list for best performance.

Workaround:

Add the NSX-T Edge VM to the NSX-V Distributed Firewall (DFW) Exclusion List by the following procedure:

Procedure

Navigate to Exclusion List settings.
- In NSX 6.4.1 and later, navigate to Networking & Security > Security > Firewall Settings > Exclusion List.
- In NSX 6.4.0, navigate to Networking & Security > Security > Firewall > Exclusion List.
Click Add.
Move the VMs that you want to exclude to Selected Objects.
Click OK.

Additional Information

Impact/Risks:

Varying degrees of network traffic impact (latency, traffic filtering) depending on the Distributed Firewall rules applied to the Edge VM's vnic. This can include management plane, control plane, and dataplane traffic impact to/from the affected Edge VM.