After an NSX upgrade, network traffic through an edge transport node is being dropped due to Distributed Firewall Rules being applied to the vNic.

Products

• VMware vDefend Firewall
• VMware NSX
• VMware NSX Firewall

Issue/Introduction

In rare cases, after an upgrade DFW rules can be applied to an Edge node even though it is included in the exclusion list, causing network traffic to be dropped.

Environment

NSX-T 3.x

NSX 4.x

vDefend Firewall

Cause

To detect  if DFW rules are applied to the Edge vNic:

1. Log into the ESXi host the edge node is residing on.
2. Run summarize-dvfilter |grep  -A2 <edge name>  and collect the "slot 2" filter name  (i.e. nic-44875694-eth0-vmware-sfw.2 )   
3. Run vsipioctl getrules -f <filter ID from command above>
4. There should be no rules returned by the vsipioctl command.  If there are rules, then continue.

      NOTE: 

       - It is normal for summarize-dvfilter to show a filter on the vNics, if the ESXi host is in a NSX prepared cluster, but there should be no rules showing in the vsipioctl command

       - Make sure to check all the interfaces on the edge as each vNic needs to be checked individually.

       - See this link for more information on using summarize-dvfilter and vsipioctl   https://docs.vmware.com/en/VMware-NSX/4.2/administration/GUID-7BCCF652-7825-4023-A5F3-A7F3B112A9F1.html

 

 

If rules are found:

1. Check the system exclusion list to see if the edge is listed.   
2. If the edge isn't in the system exclusion list, then check the User defined Exclusion list as well.

     NOTE:

        - The edge may not be in the system-excluded exclusion list if it was deployed from OVA rather than deployed from the NSX UI

 

 

If the Edge VM is NOT in either the system or user defined  exclusion lists

• Add the tag "SystemVM_NSGroup" manually to the NSX manager that is manually deployed.
• or add it to the User Defined Exclusion list and then confirm the rules are removed.

 

 

If the Edge VM IS in one of the exclusion list but still has DFW rules assigned then

1. Log into the ESXi host cli where the edge resides.
2. Run nsxdp-cli vswitch instance list and get the PortID of the vNic with the DFW rules
3. Run  net-dvs -l | less
4. Search for port <port id> where port id is from the nsxdp-cli command
5. Verify if the port definition contains the following lines.  If these exist, proceed to the Resolution

   com.vmware.port.extraConfig.security.enable = true ,    propType = CONFIG
   com.vmware.port.extraConfig.opaqueNetwork.id = xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ,   propType = CONFIG
   com.vmware.port.extraConfig.logicalPort.id = xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ,     propType = CONFIG
   com.vmware.port.extraConfig.vnic.external.id = xxxxxxxxx-xxxx ,         propType = CONFIG

Resolution

Gather logs

Please gather ESXi and NSX Manager logs before proceeding, and open a case with Support to help with root cause determination. 

 

Option 1

1. Disconnect and reconnect the edge vnic
2. Verify the rules have been removed by using vsipioctl

Option 2

1. Delete the following extraConfig lines by running the following commands on the ESXi host cli.

   net-dvs -u "com.vmware.port.extraConfig.security.enable" -p <port number> <dvs name>
   net-dvs -u "com.vmware.port.extraConfig.opaqueNetwork.id" -p <port number> <dvs name>
   net-dvs -u "com.vmware.port.extraConfig.logicalPort.id" -p <port number> <dvs name>
   net-dvs -u "com.vmware.port.extraConfig.vnic.external.id" -p <port number> <dvs name>
   
      Where
       <port number>  <--- see above on how to find that port number
       <dvs name>     <--- find this value in the first line of 'net-dvs -l'

2. Verify the rules have been removed by using vsipioctl getrules -f <filter id>