NSX-T Edge crashes with memory corruption due to ALG (FTP) traffic.

Products

• VMware vDefend Firewall

Issue/Introduction

Issue: NSX-T Edge crashes with memory corruption due to ALG (FTP) traffic. 

 

Symptoms:

• NSX-T environment with thousands of ALG(FTP) traffic sessions/ connections passing via Edge.
• Gateway firewall rule with ALG (FTP) service configured
• Edge crashes with the core dump /var/log/core/core.dp-fw-purgeX.gz
• The core file backtraces indicate the cash is due to ALG traffic.

 

 #0 __bswap16 (_x=<optimized out>) at datapath/pf/tcpip3/freebsd/amd64/machine/endian.h:188
#1 pf_alg_port_expiry (port=port@entry=0x600ffdf) at datapath/pf/pf/pf_alg.c:527
#2 0x00001cddfe3062b2 in pf_purge_expired_alg_ports (kif=kif@entry=0x65284d370380, all=all@entry=0) at datapath/pf/pf/pf_alg.c:588
#3 0x00001cddfe2bd375 in pf_purge_expired_states (kif=0x65284d370380, maxcheck=4294967295, maxcheck@entry=15000, all=all@entry=0, coreid=0) at datapath/pf/pf/pf.c:4369
#4 0x00001cddfe287d27 in dpdk_purge_state (cookie=<optimized out>, coreId=<optimized out>) at datapath/pf/pf_glue/glue.c:2106
#5 0x00001cddfe12e0f1 in firewall_sp_purge_timer_callback (timer=0x0, cb=<optimized out>) at datapath/firewall.c:7437
#6 firewall_purge_thread (args=<optimized out>) at datapath/firewall.c:7449
#7 0x00001cddfe59e0f7 in ovsthread_wrapper (aux_=0x1cde02d2f100) at edge/openvswitch/lib/ovs-thread.c:296
#8 0x0000652ead24d6db in ?? ()
#9 0x0000000000000000 in ?? ()

 

#0 __bswap16 (_x=<optimized out>) at datapath/pf/tcpip3/freebsd/amd64/machine/endian.h:188
188 datapath/pf/tcpip3/freebsd/amd64/machine/endian.h: No such file or directory.
[Current thread is 1 (LWP 23656)]
(gdb) where
#0 __bswap16 (_x=<optimized out>) at datapath/pf/tcpip3/freebsd/amd64/machine/endian.h:188
#1 pf_alg_port_expiry (port=0x600ffdf) at datapath/pf/pf/pf_alg.c:527
Backtrace stopped: Cannot access memory at address 0x67f456ad36e8

Environment

NSX-T Data Center 3.x

NSX-T Data Center 4.x

Cause

In scenarios with multiple FTP connections, the logic in the code that cleans up old connections is faulty, leading to memory corruption/crash.

 

Resolution

The fix will be included in NSX-T 9.0 and will be backported for any 4.2.x feature release after 4.2.1.