Traffic disruption when NSX-T IDPS inspects SMB traffic flows

Products

• VMware NSX

Issue/Introduction

• NSX-T Data Center 3.2.0 and 3.2.0.1
• IDPS is enabled with rules that inspect SMB traffic
• Traffic monitored by IDPS rules may be intermittently disrupted by latency or packet drops
• IDPS core files are generated on ESXi hosts /var/run/log/hostd.log

46385:2022-02-10T02:49:59.947Z: [UserWorldCorrelator] ##############us: [vob.uw.core.dumped] /usr/lib/vmware/nsx-idps/bin/nsx-idps(#######) /var/core/nsx-idps-zdump.000
46386:2022-02-10T03:00:38.398Z: [UserWorldCorrelator] ##############us: [vob.uw.core.dumped] /usr/lib/vmware/nsx-idps/bin/nsx-idps(#######) /var/core/nsx-idps-zdump.001

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

Cause

This issue is triggered when the IDPS service running on the ESXi host is not releasing the SMB transaction objects promptly, the older transaction objects continue to accumulate thereby increasing the memory consumed by the IDPS process. This leads to the engine crash and restart due to memory depletion.

Resolution

This issue is resolved in NSX-T Data Center 3.2.1 available at Broadcom Product Downloads

Workaround:
Reconfigure to ensure SMB traffic is not monitored by IDPS rules.
Alternatively put the VMs involved on the DFW exclusions list which will exclude their traffic from IDPS inspection.