NSX vDPI Service Crash on ESXi Host Transport Node Due to Bulk DNS Requests

Products

• VMware NSX

Issue/Introduction

• Traffic disruption for context-aware firewall rule objects.
• Alarm triggered in NSX UI:

  Application on NSX node [ESXi HOST TN UUID] has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team.

• ESXi host transport node logs:
  

  /var/run/log/vobd.log

  [UserWorldCorrelator] 3417085232754us: [esx.problem.application.core.dumped] An application (/usr/lib/vmware/nsx-vdpi/bin/vdpi) running on ESXi host has crashed (1 time(s) so far). A core file may have been created at /var/core/vdpi-zdump.<xxx>

• ESXi host transport node core dump file vdpi-zdump.<xxx> created in:  /var/log/core/

• ESXi host transport node shows vDPI memory usage (CurrSize) above 900MB from the following NSXCLI command:  get firewall thresholds 

 

Environment

VMware NSX 4.x

Cause

This issue occurs when multiple DNS requests are sent from the same source port causing APP_ID vector build up. The result is that vDPI crosses its memory limit before terminating and rebooting.

L7 processing failure occurs when vDPI crashes and reboots.  This results in dropped L7 firewall rule connections.

Resolution

This issue is resolved in VMware NSX 4.1.2.3, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Additional Information

Steps to remove core dump files can be found in the following article:  Application on NSX Node has Crashed Alarm