Traffic going through Edge Node is interrupted and dataplane service (dp-fp) crashes and generates core dumps
search cancel

Traffic going through Edge Node is interrupted and dataplane service (dp-fp) crashes and generates core dumps


Article ID: 318417


Updated On:


VMware NSX Networking


  • Traffic going through Edge Node is interrupted and dataplane service (dp-fp) crashes and generates core dumps.
  • Edge Node logs (syslog.log) display message(s) similar to:
#get log-file syslog.log | find core.dp-fp
<12>1 2019-12-02T05:15:36.635451+00:00 NSX - - - Core file generated: /var/log/core//core.dp-fp:1.1575263711.2688.0.11.g
  • Edge Node logs (kern.log) display message(s) similar to:
#get log-file kern.log | find "Segmentation fault occurred"
kern.log:<5>1 2019-08-28T01:00:29.224169+00:00 kernel - - - [3198001.445096] grsec: Segmentation fault occurred at 0000000000000066 in /opt/vmware/nsx-edge/sbin/datapathd[dp-fp:1:1697] uid/euid:0/0 gid/egid:124/124, parent /lib/systemd/systemd[systemd:1] uid/euid:0/0 gid/egid:0/0
  • When BGP is used, BGP neighborship goes down/up when the dataplane service crashes.
  • When static routes are used, Logical Router routing tables display "unknown" interfaces and static routes are not working as expected following dataplane service crashes.
#nsxt-edge01(tier0_sr)> get route
t0s> * [1/0] via, unknown, 05w2d02h
t0c * is directly connected, uplink-277, 12:52:30
t0c> * is directly connected, uplink-277, 12:52:30
t0s> * [1/0] via, unknown, 05w2d02h
t0s> * [1/0] via, unknown, 05w2d02h
t0s> * [1/0] via, unknown, 05w2d0


VMware NSX-T Data Center
VMware NSX-T Data Center 2.x
VMware NSX-T


This issue occurs due to an issue with the Flow cache feature (caches forwarding decision on Edge Nodes to improve performances) where segmentation fault occurs when a NULL flow entry being added to the Flow Cache tables. As a result the dataplane service on the Edge Node crashes causing a dataplane impact as described in the Symptoms.


This issue is resolved in:

VMware NSX-T Data Center 2.4.3, available at VMware Downloads.

VMware NSX-T Data Center 2.5.1, available at VMware Downloads 

VMware NSX-T Data Center 3.0, available at VMware Downloads.





To workaround this issue, disable Flow Cache on all the Edge Nodes following the steps below:
1. Login to the Edge Node using the admin account.
2. Disable Flow cache:
> set dataplane flow-cache disabled
3. Restart the dataplane server
> restart service dataplane -> (Note this will cause a brief interruption to the dataplane)
Note: this setting persists across Edge Node reboot but Flow Cache will need to be disabled on newly deployed Edge Nodes.

Confirm Flow cache is disabled:
> get dataplane flow-cache config
Example of expected output:
Enabled            : false
Mega_hard_timeout_ms: 0
Mega_size          : 0
Mega_soft_timeout_ms: 0
Micro_size         : 0

To workaround the issue where the static routes are showing as "unknown" remove and re-add the static route or edit the route and publish again, this is temporary workaround and issue may occur again so Flow Cache should be disabled to prevent re-occurrences.