NSX-T Edges segfault and causes data path disruption

Products

• VMware NSX

Issue/Introduction

NSX-T edge reports datapathd core dumps and segmentation fault is observed in the log file. If this happens to an Active NSX edge, the Edge HA process might kick in and can cause brief data path disruption. 

Edge /var/log/syslog

2024-05-17T15:35:43.622Z <edge-node> kernel - - - [10285195.004684] grsec: Segmentation fault occurred at 0000000000008050 in /opt/vmware/nsx-edge/sbin/datapathd[dp-fp:2:2711580] uid/euid:0/0 gid/egid:124/124, parent /opt/vmware/edge/dpd/entrypoint.sh[entrypoint.sh:2711404] uid/euid:0/0 gid/egid:124/124
2024-06-11T03:41:36.601Z <edge-node> kernel - - - [12402147.867949] grsec: Segmentation fault occurred at 0000000000000004 in /opt/vmware/nsx-edge/sbin/datapathd[dp-fp:7:977880] uid/euid:0/0 gid/egid:124/124, parent /opt/vmware/edge/dpd/entrypoint.sh[entrypoint.sh:977762] uid/euid:0/0 gid/egid:124/124
2024-06-27T03:36:36.990Z <edge-node> kernel - - - [13784114.173050] grsec: Segmentation fault occurred at 0000000000000004 in /opt/vmware/nsx-edge/sbin/datapathd[dp-fp:7:1535762] uid/euid:0/0 gid/egid:124/124, parent /opt/vmware/edge/dpd/entrypoint.sh[entrypoint.sh:1535605] uid/euid:0/0 gid/egid:124/124
2024-08-08T03:39:05.347Z <edge-node> kernel - - - [17412693.233615] grsec: Segmentation fault occurred at 0000000000000004 in /opt/vmware/nsx-edge/sbin/datapathd[dp-fp:0:1723403] uid/euid:0/0 gid/egid:124/124, parent /opt/vmware/edge/dpd/entrypoint.sh[entrypoint.sh:1723364] uid/euid:0/0 gid/egid:124/124

 

Edge /var/log/kern.log

2023-06-23T07:10:16.958Z <Edge-node> kernel - - - [1514204.571626] dp-fp:1[9868]: segfault at 4 ip 00001464583439c1 sp 0000743387dd5410 error 4 in datapathd[146458000000+15f1000]
2023-06-23T07:10:16.960Z <Edge-node> kernel - - - [1514204.571639] Code: 16 48 8b 57 18 48 8b 8f c8 00 00 00 48 83 c0 04 f6 c2 02 48 8b 34 c1 74 38 8b 47 2c 48 83 8f c0 00 00 00 10 89 87 dc 00 00 00 <8b> 46 04 83 e0 03 83 f8 03 75 3c 48 89 df e8 7c bb fe ff 48 83 c4

 

Edge directory /var/log/core has similar core dumps:

-rw-r--r--  1 root root 1.3G May 17 15:40 core.dp-fp:2.1715960143.2711437.0.11.gz
-rw-r--r--  1 root root 1.3G Mar 27 03:08 core.dp-fp:4.1711508622.9146.0.11.gz
-rw-r--r--  1 root root 1.3G Jun 11 03:46 core.dp-fp:7.1718077296.977800.0.11.gz
-rw-r--r--  1 root root 1.3G Jun 27 03:42 core.dp-fp:7.1719459396.1535647.0.11.gz

Environment

VMware NSX-T Data Center

 

Cause

This is caused by a software issue in NSX-T 3.x

Resolution

This issue is fixed in NSX 4.2.0. Affected customers should upgrade to this version. 

There is currently no other workaround available. 

If an immediate upgrade is not possible, please open a Broadcom Support case referencing this KB.