Enabling Distributed Firewall on KVM impacts connection handling performance

Products

• VMware vDefend Firewall

Issue/Introduction

Symptoms:
When establishing a large number of connections between virtual machines, you may experience these symptoms:

• Netperf may report errors similar to:
  
  shutdown_control: no response received errno 104
  
  
• The issue seems to manifest itself when default firewall rules have been configured
• With the default firewall rule configured, the issue do not manifest itself with a lighter load (For example: with 1 virtual machine x 64 sessions, or 8 virtual machines x 4 sessions)
• The issue do not manifest itself if firewall rules are not configured (For example: Logical switches are put in the firewall exclusion list).
• In the /var/log/syslog or /var/log/messages file, you see entries similar to:
  
  Apr 26 11:45:44 prmh-nsx-perf-server149 kernel: [1625289.950872] net_ratelimit: 239 callbacks suppressed
  Apr 26 11:45:44 prmh-nsx-perf-server149 kernel: [1625289.950875] nf_conntrack: table full, dropping packet
  Apr 26 11:45:44 prmh-nsx-perf-server149 kernel: [1625289.958436] nf_conntrack: table full, dropping packet
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T 1.0.x
VMware NSX-T

Cause

This issue occurs due to the fixed size of the conntrack table.

Resolution

This is a known issue affecting VMware NSX-T 1.0.x.

Currently, there is no resolution.

To work around this issue, set the nf_conntrack_tcp_timeout_time_wait to 0 by running this command:

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=0

Additional Information