NSX Firewall (Gateway FW or Distributed FW) Rule Impacted Due to Missing Effective Member IP(s) from NSGroup

Products

• VMware NSX

Issue/Introduction

• NSX Firewall (GW FW or DFW) is dropping flows that are expected to be allowed or allowing flows that are expected to be dropped.
• NSX IP discovery is configured with Duplicate Detection, ARP Snooping and VMware Tools enabled.
• NSX IP discovery is configured with Trust on First Use (TOFU) disabled.
• Logical-port IP has a realized binding with discovered type VM Tools only.
• Logical-port IP is not realized and is not present as an effective member for the NSGroup being used by the impacted FW rule.
• Traffic for the above logical-port IP is impacted.

Environment

VMware NSX-T Data Center

VMware NSX

Cause

When an NSX logical-port IP is discovered by ARP Snooping and VMware Tools at the same time, the IP Discovery duplicate detection logic incorrectly considers the ARP Snooping and VMware Tools bindings to be the same because they share the same binding timestamp, VLAN ID, MAC address and IP address.  As a result, the VMware Tools entry is not added to the TreeMap or relevant dynamic NSGroups. 

When ARP expires, the ARP Snooping entry for this IP is also removed, meaning there are no realized bindings or effective members for this IP in the NSGroups.  FW rules that are using these NSGroups may then be impacted by traffic disruption.

 

Resolution

This is a known issue impacting VMware NSX.

 

Workaround

The following options may be used as temporary workarounds but should be implemented based on environmental requirements:

1. Add static entry for impacted IP to impacted FW rule(s).
   
   
2. Configured NSX IP discovery with Trust on First Use (TOFU) enabled:
   • Enabling TOFU may not be possible in all enviornments ie. if VM HA is in use:
     Configure and apply NSX-T Segment IP discovery profile when using high availability (HA) for Virtual Machines
     
     
3. When in an impacted state the NSX UI will show a single realized binding (ie. Discovery Type VMware Tools).  To restore traffic flow, perform the following steps:

1. 1. View realized bindings for the impacted logical-port from the NSX UI:
      Networking > Segment (where impacted logical-port is connected) > Click number under Ports / Interfaces - Expand impacted Logical-Port > Expand Address Bindings  - Click view under Realized Binding
   2. Select the single realized binding > Click 'Copy To' - Ignore Binding.
   3. Delete the ignored binding and save.
   4. Realized bindings will automatically update with new VM Tools and ARP Snooping entries, restoring impacted traffic flow.