Distributed Firewall (DFW) packets hitting Default Rule instead of previous Rule allowing/blocking designated traffic

Products

VMware vDefend Firewall

Issue/Introduction

Symptoms:

In examining Distributed Firewall (DFW) behavior, you may see some packets having a source, destination, and protocol (service) defined in a configured rule hitting the Default rule at the very end of the firewall rule list.

In the dfwpktlogs or in LogInsight, you may see that the SYN packets and the ACK packets are being processed by the configured rule allowing/blocking the defined traffic.

But for the same traffic, you see RST and FIN ACK packets hitting the default block/allow rule, meaning that it is not hitting the previously-configured rule.

Thus, it appears that the DFW is not consistently processing packets for traffic defined in the particular rule.

However, this is the expected behavior of the DFW Stateful Firewall for TCP Packets:

For rules with Protocol TCP, the TCP Flags on the rule, by default is set to SYN and hence when a TCP packet arrives, its TCP Flags are also compared with the rule apart from the standard 5 tuples. However, rules that do not specify a protocol (service ANY), can match any TCP packets with any combination of flags.

Thus, if TCP protocol is defined in a rule allowing traffic between Source A and B, a flow (or state) will be generated/matched against this rule only upon receipt of the first SYN packet. This flow will remain open, allowing traffic to flow freely between A and B, as rest of the flow packets will match this created state.

However, when either A or B terminate the connection and the flow is deleted from DFW, a late packet such as RST or a FIN ACK or a PUSH ACK packet may arrive at the firewall from either side. As the flow has already been terminated, the DFW treats this packet as a new flow and must commence comparing the traffic with its configured rules. Since the RST or the FIN ACK do not have the SYN flag set, the packet does not match a specific TCP rule and these leftover packets may hit the default rule or any other rule lower in the priority than the specific TCP rule from with the initial session was anchored (or the first rule with "any" as the service - usually the default rule).

Again, this is expected behavior and will have no impact on active connections or traffic content. For more information, see the Distributed Firewall section of the NSX Administration Guide.

Environment

VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.0.x

Resolution

To see what flows are active for a particular virtual machine, ensure that Flow Monitoring is enabled. For more information, see the Configure Flow Monitoring section of the NSX Administration Guide.

Open an SSH Session to the ESXi host where the virtual machine is running.
Run the summarize-dvfilter | grep -A20 command to retrieve the virtual machine DFW Filter name.
Run the vsipioctl getflows -f command to see the active or current flows associated with that filter or virtual machine.

To verify the rules ascribed to the filter or vNIC, run the vsipioctl getrules -f command. From this output, you can verify that the configured rule(s) is being applied to the virtual NIC as expected.

There are multiple ways to verify and to confirm that the firewall is working as intended.

You could add a deny rule with the service as ANY just below the allow rule with the same Source and Destination. In the logs, you should see the non-SYN packets hitting the new Deny rule instead of the Default rule.
In addition, you can use Traceflow to illustrate how the DFW works.

In the example below, a Firewall rule is set up allowing TCP traffic between web-01a and web-02a. The default rule is set to Block. In the Advanced options of Traceflow, you can specify a particular TCP flag for the monitored packet.

In the first example, a SYN packet is specified. The SYN packet is delivered and hits the Allow rule before going to the Default Block rule.

In this example, an RST packet is sent. In this case, the packet is dropped at the firewall, since the firewall only matches SYN packets for defined rules.