NSX-T Edges segmentation fault with data path disruption

Products

• VMware NSX

Issue/Introduction

• NSX-T edge reports datapathd core dumps and segmentation fault is observed in the log file.
• If this happens to an Active NSX edge, the Edge HA process might kick in and can cause brief data path disruption. 
• Log entries similar to the below will be observed on the Edge Node in /var/log/syslog
  

  2024-05-17T15:35:43.622Z <edge-node> kernel - - - [10285195.004684] grsec: Segmentation fault occurred at 0000000000008050 in /opt/vmware/nsx-edge/sbin/datapathd[dp-fp:2:2711580] uid/euid:0/0 gid/egid:124/124, parent /opt/vmware/edge/dpd/entrypoint.sh[entrypoint.sh:2711404] uid/euid:0/0 gid/egid:124/124

• Log entries similar to the below will be observed on the Edge Node in /var/log/kern.log

  2023-06-23T07:10:16.958Z <Edge-node> kernel - - - [1514204.571626] dp-fp:1[9868]: segfault at 4 ip ################ sp ################ error 4 in datapathd[146458000000+15f1000]
  
  2023-06-23T07:10:16.960Z <Edge-node> kernel - - - [1514204.571639] Code: 16 48 8b 57 18 48 8b 8f c8 00 00 00 48 83 c0 04 f6 c2 02 48 8b 34 c1 74 38 8b 47 2c 48 83 8f c0 00 00 00 10 89 87 dc 00 00 00 <8b> 46 04 83 e0 03 83 f8 03 75 3c 48 89 df e8 7c bb fe ff 48 83 c4

• Edge directory /var/log/core has core dumps similar to the below:

  -rw-r--r--  1 root root 1.3G May 17 15:40 core.dp-fp:2.1715960143.2711437.0.11.gz

Environment

VMware NSX-T Data Center

 

Cause

A corruption in mbuf leads to invalid ifp aka interface or port which in-turn causes datapath to crash during input packet processing.

Resolution

This issue is resolved in VMware NSX 4.2.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

If you believe you have encountered this issue and are unable to upgrade, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.