dp-fw-dispatch process crash on ESXi host causes an NSX alarm, Application on NSX node <hostname> has crashed

Products

• VMware NSX

Issue/Introduction

• You are running VMware NSX 4.x.
• In the NSX manager UI, the below alarm is generated with the following details:

Application on NSX node <NSX-Edge node> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact Broadcom Support team. 

• On the NSX Edge node, In the log file /var/log/syslog you see entries similar to the following:

nsx-edge-01 NSX 1466 - [nsx@6876 comp="nsx-edge" subcomp="nsx" username="nsx" level="CRITICAL" eventFeatureName="infrastructure_service" eventType="application_crashed" eventSev="critical" eventState="On" entId="91g26v51-####-####-####-e6e50a998777"] Application on NSX node <NSX-Edge node> has crashed. The number of core files found is 1. Collect the Support Bundle including core dump files and contact VMware Support team.

• On the mentioned NSX Edge node, you see the following core dump generated:

/var/log/core/core.dp-fw-dispatch.xxxx.gz

• There may be unexpected HA failover with datapath impact at the time of the above core dump.

Environment

VMware NSX

Cause

A race condition when handling certain update messages relating to nest-db on the edge node can cause the fw-dispatch service to crash leading to a node failover. These update messages can be related to IDS, GWFW and Bridge Firewall.

However these services are not required to be configured for this issue to potentially occur as IDS service can still trigger update messages to edge nodes relating to signature update availability without IDS being in use.

Resolution

This issue is resolved in VMware NSX 4.1.2.4
This issue is resolved in VMware NSX 4.2.0