IDPS event loss observed on the NSX Manager UI when IDPS events hit rate limitation

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

When IDPS events hit rate limitation i.e. when the rate at which IDPS alerts are received is higher than what can be processed by the NSX Manager, some events may be dropped.

Environment

vDefend Firewall with ATP 9.1 with IDPS enabled / configured

Cause

The issue occurs when there is a very high number of alerts sent to the NSX Manager. As the manager cannot process these alerts at the incoming rate, they get queued. The queue may then outgrow the allocated resources and cause instability to the IDPS service. To avoid this situation, NSX Manager drops a subset of IDPS events and pcaps to protect the system from instability. The drops happen based on criticality - the non-critical events and pcaps are dropped before the critical ones.

This is caused by either an undersized manager appliance form-factor or is a direct result of a large-scale security event that is storming the infrastructure.  

Logs pertaining to current memory usage and drops can be found on the NSX Manager at /var/log/idps-reporting/idps.log

Resolution

Investigate the source of high number of IDPS events. Additionally, please collect the NSX Manager support bundles and raise a support ticket with the Broadcom Support Team.