Alarm for NSX IDPS Engine Memory Usage high

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Title: "Alarm for NSX IDPS Engine Memory Usage high "

Event ID: 

• • distributed_ids_ips.nsx_idps_engine_memory_usage_high
  • distributed_ids_ips.nsx_idps_engine_memory_usage_high_on_dpu
  • distributed_ids_ips.nsx_idps_engine_memory_usage_medium_high
  • distributed_ids_ips.nsx_idps_engine_memory_usage_medium_high_on_dpu
  • distributed_ids_ips.nsx_idps_engine_memory_usage_very_high
  • distributed_ids_ips.nsx_idps_engine_memory_usage_very_high_on_dpu

Added in release: 3.1.0
Alarm Description
Purpose: Limit reached for memory usage of the IDPS engine.
Impact: There may not necessarily be any visible impact. The alarms serve as a warning that traffic may not be inspected for intrusions if memory usage is left unchecked.

Environment

VMware NSX

Resolution

Maintenance window required for remediation: no
Steps to resolve:

• Fine-tune the IDPS rules so that the datapath engine is subject to less traffic.
• Review the IDPS engine stats using the command below to see the application layer protocol-related statistics. This can help determine the type of traffic processed by the engine, which in turn can help determine the type of workload VMs to investigate.
  • NSX 4.2 and earlier (non-SCRX)
    Suricata app layer memory stats: "nsxcli -c get ids engine stats"
    To see memory consumed by the IDPS process
    1. vsish -e set /sched/groupPathNameToID host vim vmvisor nsx-idps, this returns a <process ID>
    2. memstats -r group-stats -s name:max:consumed -u mb -g <process ID>
  • NSX 4.2.1 and later (SCRX)
    Service Datapath memory stats
    /opt/vmware/nsx-cli/bin/nsx-appctl -t /var/run/vmware/scx/sdp.ctl sdp/get/mem/stats
    IDPS engine stats
    /opt/vmware/nsx-cli/bin/nsx-appctl -t /var/run/vmware/scx/sdp.ctl sdp/get/service/idps/stats
• You can further look at per filter packet incoming/outgoing statistics using the vsish commands
  • vsish -e get /net/portsets/DvsPortset-0/ports/<PORT_NUMBER>/inputStats and look at DVFILTER_VNIC_IN_GUEST stats
  • vsish -e get /net/portsets/DvsPortset-0/ports/<PORT_NUMBER>/outputStats and look at DVFILTER_VNIC_OUT_GUEST stats
• You can additionally use other tools like VMware vRealize Network Insight vmw-vrni-solution-brief.pdf to gain better insights of the deployment.