NSX Advanced Firewall IDPS Performance Tuning Considerations

Products

• VMware Cloud on AWS
• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall
• VMware NSX Firewall

Issue/Introduction

This article focuses on the Distributed Intrusion Detection & Prevention (D-IDPS) component of the NSX Security Platform. The guidance documented here provides recommended practices to allow for DIDPS to be deployed in a scalable and performant manner.

Environment

NSX-T 

Resolution

What is IDPS?

IDS is fundamentally Pattern Matching for signs of known attacks. NSX IDPS inspects traffic that is specifically being allowed IN or OUT  via a Distributed Firewall policy.

​NSX Distributed IDPS is a control point that is situated logically at every Virtual NIC (vNic) of all Virtual Machines (VMs).

Why is it important to fine tune IDPS?

A key aspect of any successful IDPS deployment is the tuning of policy design along with patterns or signatures to align with workload profiles and security requirements. This will reduce latency, the computational overhead for traffic analysis and will also assist in reducing the number of false positives (noise) in the environment. 

What are some of the General recommended practices for a scalable and performant D-IDPS implementation ?

When building a D-IDPS policy, you should start by limiting the attack surface using the Distributed Firewall (DFW). When a base security posture has been established, D-IDPS can be used to further enhance the security posture.

Best Practices for IDPS Performance

Compute

Before enabling D-IDPS on a node or cluster, it is best to evaluate the clusters and host resource utilisation using metrics such as Host CPU usage and CPU RDY% time for virtual workloads and make sure there is enough resources to support a minimum 5 worker threads and 1 receiver thread required by the IDPS Engine.  You can use tools such as vROPS to view the existing vCPU:pCPU oversubscription ratio.

General recommendations for ideal IDS/IPS performance:

• Reduce amount of traffic redirected to IDPS
• Configure Directionality appropriately
• Migrate workloads across hypervisors based on traffic loads

 

Rules

• Start with the Security Requirements. Where is IDPS needed ?
• First, implement segmentation to limit the attack surface
• Avoid configuring IDPS rules with 'Any' for source, destination, and service fields. IDS/IPS is resource-intensive and not intended for handling  all types of traffic in a data center.
• Exempt traffic that doesn’t need IDPS or cannot be inspected. You can use the negate-option with a security group to exclude particular workloads that don’t need IDPS inspection
  • Backup Traffic
  • Encrypted Flows
  • N-S traffic that has already been inspected
• Use Applied-To and optionally SRC/DST to only apply IDPS to workloads or flows that need to be inspected.

Directionality

• If throughput is a concern, consider single directionality (IN or OUT) to limit traffic redirected to IDPS engine

 For Server workloads:

– IN direction can be sufficient if a proper segmentation policy has been applied to block outbound (external) flows and if all communication is inbound to workloads protected by D=IDPS

– For allow outbound (external) communication, IDPS in OUT or IN/OUT direction can be applied in a granular rule 

For VDI workloads:

– OUT direction can be sufficient if a segmentation policy has been applied which blocks inbound service access to VDI desktops

• Use directionality with care. Directionality is stateful and effectively determines when a flow entry (for redirection to IDPS) is created.

The default option (IN and OUT) guarantees that IDPS is applied to all traffic that matches the rule, but the same flow is redirected twice if both SRC/DST are a VM with IDPS enabled.

Note: When using a single direction, flows initiated in the other direction are not subject to IDPS inspection!  Changing the direction to “IN” only will also result in C2 traffic from a compromised workload not being inspected!

Profiles

• Start with Detect-only mode and move to detect and prevent after initial deployment and tuning
• Start with a broad coverage profile and wide application
  • I.e. CVE: > 7.0
  • I.e. Attack-Target: Client-Endpoint, Server and Client for EUC
• Optionally, over time Include only relevant signatures in a profile and apply profile to specific workloads
  • Attack target: DNS Server, AD Server, Web Server
  • Affected product