CyPerf DNS attack profile causes IDPS core dumps

Products

• VMware vDefend Firewall

Issue/Introduction

• When running the CyPerf tool, a core dump is generated from a third-party library – Suricata. This happens when processing DNS traffic (or a mix of DNS traffic and other traffic types), for which there are no settings in the library to restrict memory usage. 
• IDPS memory consumption will throw alarms if it is too high. Alarms will also be thrown if the IDPS process crashes.
• To check the current consumed memory by IDPS on an ESXi host:
  • Run command on host CLI to obtain ID (used in Step 2):
    
    vsish -e set /sched/groupPathNameToID host vim vmvisor nsx-idps
    10######


  • Run command:
    
    memstats -r group-stats -s name:max:consumed -u mb -g <ID from Step 1>

 GROUP STATS: Tue Aug 15 15:00:38 2023
 -------------------------------------
   Start Group ID   : 10XXXXXX
   No. of levels    : 12
   Unit             : MB
   Inclusion filter : (all)
   Exclusion filter : (none)
   Selected columns : gid:name:max:consumed

-----------------------------------------------------------
     gid                         name        max   consumed
-----------------------------------------------------------
10######                  nsx-idps       1024       1002      <----------------- if this value is close to 1024 mb, IDPS process in memory is oversubscribed and likely to crash
…
...
    
    

Resolution

This is a known issue affecting VMware NSX. There is currently no resolution.

Workaround:
IDPS oversubscription feature can be used to bypass/drop traffic when there is high load.
IDPS rules can be created to exclude DNS traffic from IDPS inspection.

Additional Information

Impact/Risks:
When the DNS out of memory error is hit, the IDPS process core dumps and is restarted by the watchdog process. During this time, IDPS protection is lost.