NSX Edge node Dataplane service Crashes and core dump file core.dp-ipc.gz is generated

Products

• VMware NSX

Issue/Introduction

In VMware NSX 4.2.X.X, a coredump was generated from the NSX edge node in the path: /var/log/core/core.dp-ipc22:<process-ID.timestamp>.gz

A similar log entry maybe found in the NSX edge syslog:

2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] vrfid 65 id 0x0401dbfc313e3590 state op error: TCP x.x.x.x:46148 gwy x.x.x.x:46148 ext x.x.x.x:1029 seq [lo=3048496512 hi=3048496514 win=1024 modulator=0]
 [lo=0 hi=1 win=1 modulator=0] 2:0 p 0x763acf593740 kif 0x76397f74b9c0
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] pf_state_list_op_error:error in op:REMOVE for state:0x763acf593740 id:0401dbfc313e3590 state_list:0x76397f74d830 target_list:(nil) core_id:3 local_flags:1 state_flags:65671
timeout:22 ref_flags:0 kif:0x76397f74b9c0 refcount:1 lookup_cnt:0 rule_id:0 sync_flags: 17409 lb_flags: 0
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] pf_state_list_op_error:state_list:0x76397f74c030 closing_state_list:0x76397f74e030 unlink_state_list:0x76397f74d830 purge_state_list:0x76397f74c830 purge_state_clean_list:0x76397f74d030
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] vrfid 65 id 0x0601dbfc313a7438 state op error: TCP x.x.x.x:53936 gwy x.x.x.x:53936 ext x.x.x.x:49999 seq [lo=1532271355 hi=1532271357 win=1024 modulator=0
] [lo=0 hi=1 win=1 modulator=0] 2:0 p 0x763af686a300 kif 0x76397f74b9c0
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] pf_state_list_op_error:error in op:REMOVE for state:0x763af686a300 id:0601dbfc313a7438 state_list:0x76397f74d830 target_list:(nil) core_id:3 local_flags:1 state_flags:65671
timeout:22 ref_flags:0 kif:0x76397f74b9c0 refcount:1 lookup_cnt:0 rule_id:0 sync_flags: 17409 lb_flags: 0
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] pf_state_list_op_error:state_list:0x76397f74c030 closing_state_list:0x76397f74e030 unlink_state_list:0x76397f74d830 purge_state_list:0x76397f74c830 purge_state_clean_list:0x76397f74d030
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] vrfid 65 id 0x0401dbfc313e2b05 state op error: TCP x.x.x.x:53936 gwy x.x.x.x:53936 ext x.x.x.x:20031 seq [lo=1532271355 hi=1532271357 win=1024 modulator=0
] [lo=0 hi=1 win=1 modulator=0] 2:0 p 0x763acde25180 kif 0x76397f74b9c0
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] pf_state_list_op_error:error in op:REMOVE for state:0x763acde25180 id:0401dbfc313e2b05 state_list:0x76397f74d830 target_list:(nil) core_id:3 local_flags:1 state_flags:65671
timeout:22 ref_flags:0 kif:0x76397f74b9c0 refcount:1 lookup_cnt:0 rule_id:0 sync_flags: 17409 lb_flags: 0
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-ipc22" level=
"INFO"] pf_state_list_op_error:state_list:0x76397f74c030 closing_state_list:0x76397f74e030 unlink_state_list:0x76397f74d830 purge_state_list:0x76397f74c830 purge_state_clean_list:0x76397f74d030
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="FATAL"] PANIC in dpdk_panic():
2025-11-12T09:38:43.965Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="FATAL"] assert failed
2025-11-12T09:38:43.096Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net 837dc6bcff47 10089 - -  2025-11-12T09:38:43Z datapathd 11902 intel-rte tname="dp-ipc22" [FATAL] PANIC in dpdk_panic():
2025-11-12T09:38:43.529Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net datapath-systemd-helper 11822 - -  2025-11-12T09:38:43Z datapathd 11902 intel-rte tname="dp-ipc22" [FATAL] PANIC in dpdk_panic():
2025-11-12T09:38:43.582Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net datapath-systemd-helper 11822 - -  2025-11-12T09:38:43Z datapathd 11902 intel-rte tname="dp-ipc22" [FATAL] assert failed
2025-11-12T09:38:43.621Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net 837dc6bcff47 10089 - -  2025-11-12T09:38:43Z datapathd 11902 intel-rte tname="dp-ipc22" [FATAL] assert failed
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WA
RN"] 26: [/lib/x86_64-linux-gnu/libc.so.6(+0x126c50) [0x76482acfdc50]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WA
RN"] 25: [/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76482ac6bac3]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WA
RN"] 24: [/opt/vmware/nsx-edge/sbin/datapathd(+0x8756c7) [0x1592d64c36c7]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WA
RN"] 23: [/opt/vmware/nsx-edge/sbin/datapathd(+0x218719) [0x1592d5e66719]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WA
RN"] 22: [/opt/vmware/nsx-edge/sbin/datapathd(+0x2774b4) [0x1592d5ec54b4]]2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 11: [/opt/vmware/nsx-edge/sbin/datapathd(+0x29bef9) [0x1592d5ee9ef9]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 10: [/opt/vmware/nsx-edge/sbin/datapathd(+0x29bebc) [0x1592d5ee9ebc]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 9: [/opt/vmware/nsx-edge/sbin/datapathd(+0x44926b) [0x1592d609726b]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 8: [/opt/vmware/nsx-edge/sbin/datapathd(+0x435650) [0x1592d6083650]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 7: [/opt/vmware/nsx-edge/sbin/datapathd(+0x580cb4) [0x1592d61cecb4]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 6: [/opt/vmware/nsx-edge/sbin/datapathd(+0x5e27be) [0x1592d62307be]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 5: [/opt/vmware/nsx-edge/sbin/datapathd(+0x5e0da6) [0x1592d622eda6]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 4: [/opt/vmware/nsx-edge/sbin/datapathd(+0x57f015) [0x1592d61cd015]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 3: [/opt/vmware/nsx-edge/sbin/datapathd(+0x6220d0) [0x1592d62700d0]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 2: [/lib/librte_eal.so.22(__rte_panic+0xc5) [0x764833c2612f]]
2025-11-12T09:38:43.966Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 11902 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" tname="dp-ipc22" level="WARN"] 1: [/lib/librte_eal.so.22(rte_dump_stack+0x32) [0x764833c4d4a2]]
2025-11-12T09:38:44.152Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 2728772 - [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="INFO"] Core dump generation received by process: 11902 [dp-ipc22]
2025-11-12T09:38:44.153Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 2728772 - [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.dp-ipc22.1762940323.11902.0.6.gz
2025-11-12T09:38:44.265Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 1894483 - [nsx@6876 comp="nsx-edge" subcomp="nsx-sha" username="nsx-sha" level="INFO"] [name:daemon_health_monitor][id:217c1efc-c0b1-41ca-a65a-e783f7303b6f] new agent_state: {'NSX_OPSAGENT': {'state': 1, 'last_update_time': 10238058}, 'last_report_time': 22886208, 'last_report_reason': 'fullsync'}, should report the state change
2025-11-12T09:38:44.266Z xx-xxxx-1-xxx-nsx-edge04-node01.xxx.ip-xxxx.net NSX 1894483 - [nsx@6876 comp="nsx-edge" subcomp="nsx-sha" username="nsx-sha" level="INFO"] Dhreport - reporting, tn_agent_status:{"oid":{"left":"5221212016464512695","right":"12752145131346251821"},"status":[{"type":"TN_AGENT","tn_agent_status":{"threat_state":{},"agent_state":{"total_status":"UP","up_count":1,"down_count":0,"agent_status":[{"type":"NSX_OPSAGENT","status":"UP","connection_status":[{"name":"opsagent-proxy-connection","status":"UP"}]}],"degraded_count":0}}}]}

Environment

NSX 4.2.X.X

Cause

This issue occurs when Tier1 gateways is moved to different edge cluster under high connection state load, causing race condition between IPC cleanup thread and purge threads. This results in dataplane crash and core dump generation.

Resolution

This is a known issue caused by moving the T1 router between edge clusters during periods of high active connections. To avoid the problem, ensure that T1 router migrations are not performed during peak traffic. Instead, carry out any movement during low-traffic maintenance windows. A permanent fix is planned for an upcoming NSX release, and customers are advised to upgrade to the version containing the fix once it becomes available.