Performance degradation and high network latency experienced on ENS poll mode enabled virtual Machines

Products

• VMware NSX
• VMware Telco Cloud Platform

Issue/Introduction

Environments running workloads with ENS Poll Mode enabled eperience any of the below behaviours:

• Applications experience low data transfer speeds below the expected benchmark or hardware capabilities.
• Intermittent or sustained high latency and dropped packets during periods of high traffic volume or sudden burst of traffic.
• There is an unexpected spike in CPU utilization on some of the Virtual Machines.
• The underlying virtual machines areprotected through NSX DFW (Distrubuted Firewall). 

Environment

• NSX 4.1.2.1
• TCP 5.0

Cause

• All newly provisioned virtual machines that are connected to an NSX segment are automatically proetcted  by the NSX DFW by default. 
• When virtual machines are protected by NSX DFW, the ESXi host inspects the packets against the firewall rules.
• The overhead introduces a slight negligible latency which may not match the throughput requirments of applications that handle high frequency network workloads.
• If the overhead is not balanced against the expected strict latency and throughput requirements of the applications there may be a battleneck causing delays in packet processing. 
• Specifically, when VMs have ENS poll mode enabled, the DFW packet inspection interrupts the accelerated CPU polling cycle and may divert accelerated traffic to the standard ESXi kernel slowpath. 

Resolution

• Exclude the virtual machines with ENS poll mode from NSX DFW protection by creating a NSX Security group and configuring exclusion rules. Please refer to Performance best practices for data plane intensive CNFs
  
  

Option 1:

Create Dedicated Logical Switches (Segments) or Ports and add it to the firewall exclusion list. 

1. Log into the NSX Manager UI.

2. Navigate to Security > Distributed Firewall.

3. Click the Actions dropdown menu on the firewall policy window and select Exclusion List.

4. Choose either logical switches or logical potrs as per the infrastructure design:

   • Select Logical Switches (Segments) to add the entire network segment hosting the workloads.

   • Select Logical Ports to target the specific virtual ports bound to the impacted virtual machines.

5. Click Save / Apply.

Any current or future virtual machine connected to these excluded switches or ports will automatically bypass the NSX DFW firewall inspection.

Option 2:

Automate Exclusion using Infrastructure Tags 

1. Create a tag at the vCenter Server Create a Tag, ensure target virtual machines are assigned a designated tag identifier (e.g., Category: Workload-Type, Tag: <Tag name>).

2. Log into NSX Manager and navigate to Inventory > Groups.

3. Click Add Group.

4. Click Set Members and stay on the default Membership Criteria tab.

5. Click Add Criterion and set the rule: Object: Virtual Machine | Property: Tag | Operator: Equals | Value: <Tag name>.

6. Click Apply, then Save.

7. Return to Security > Distributed Firewall > Actions > Exclusion List.

8. Under the Groups tab, add your newly created dynamic group and click Save.