Dataplane Service crashed on NSX Edge node or Bare Metal
search cancel

Dataplane Service crashed on NSX Edge node or Bare Metal

book

Article ID: 432198

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • The dataplane service crashed on an NSX Edge node or Bare Metal.
  • The NSX Edge node or Bare Metal may also have been temporarily unreachable on its Management interface.
  • The datapath service automatically restarts and the impact is resolved.
  • This can occur when editing the Active/Standby Tier 1 logical router and disabling Auto Allocate Edges, in order to manually assign edges in the cluster.
  • A core dump was generated for datapathd and/or dp-ipc.
    Sample list of core dumps with command: get core dumps
    > get core dumps
    core.datapathd.##########.gz
    core.dp-ipc56.##########.gz
  • Log lines similar to the below are encountered in /var/log/syslog
    Sample logs from /var/log/syslog
    datapathd 2507612 intel-rte tname="dp-ipc56" [FATAL] PANIC in dpdk_panic():
  • Log lines similar to the below are encountered in /var/log/syslog
    Sample logs from /var/log/syslog
    NSX FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc56" level="INFO"] pf_state_list_op_error:error in op:REMOVE for state:0x############ id:0000000000000000 state_list:(nil) target_list:(nil) core_id:6 local_flags:1 state_flags:65667 timeout:22 ref_flags:0 kif:0x############ refcount:0 lookup_cnt:0 rule_id:0 sync_flags: 1025 lb_flags: 0
  • Log lines similar to the below are encountered in /var/log/nvapi/api_server.log 
    Sample logs from /var/log/nvapi/api_server.log
    napi.root.node.edge_utils ERROR edge-appctl failed for service dataplane: out:, err:2026-03-13T14:35:11Z edge-appctl 77699 jsonrpc [WARN] unix:/var/run/vmware/edge/dpd.ctl: receive error: Connection reset by peer
    /var/run/vmware/edge/dpd.ctl: transaction error (Connection reset by peer)
    napi.root.node.utils ERROR Cannot find socket file: /var/run/vmware/edge/dpd.ctl
    napi.root.node.edge_utils ERROR edge-appctl failed for service dataplane: out:, err:2026-03-12T14:32:37Z edge-appctl 3356100 u
    nixctl [WARN] failed to connect to /var/run/vmware/edge/dpd.ctl

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

The root cause is a race condition between the state-table shutdown (during service router deletion) and the internal purge mechanism.

A stale cursor continues to point to unlinked states, leading to a double-decrement of NAT rule counters. This results in a memory counter underflow (pf_state_list_op_error), which triggers the PF_VERIFY assertion failure and forces the nsx-dataplane service to crash to prevent further corruption.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

The datapath service will automatically attempt to restart after the core dump is generated, restoring the communication.

If the NSX Edge node or Bare Metal remains in an inconsistent state or the nsx-dataplane fails to initialize, reboot it.

Powered by Wolken Software