ESXi Host encounters PSOD due to a multi-thread race condition in NSX Distributed Firewall (DFW)

Products

• VMware vDefend Firewall

Issue/Introduction

In rare scenarios, an ESXi host may experience a PSOD due to a multi-thread race condition in the Distributed Firewall (DFW) module (nsxt-vsip) when retrieving firewall flow state data.

• The faulting module indicated on the PSOD screen is nsxt-vsip.

• The crash occurs when a closed Application Level Gateway (ALG) flow is retrieved by nsx-exporter at the exact time the DFW purge thread (VSIP PF Purge Thread) is deleting internal state objects.

• The PSOD and stacktrace will appear in vmkernel.log with a backtrace similar to:

(gdb) where
#0  0x################ in pfioctl (kif=kif@entry=0x################, dev=dev@entry=0x0, cmd=cmd@entry=############, addr=addr@entry=0x################ "########", flags=<optimized out>, flags@entry=0, td=td@entry=0x0) at datapath/esx/modules/vsip/vsip_pf/pf/pf_ioctl.c:4510
#1  0x################ in VSIPConversionGetConnectionStat (kif=0x################, msg=0x################, msgLen=<optimized out>, result=0x################) at datapath/esx/modules/vsip/vsip_pf/pf_vmk/msg2pf.c:6974
#2  0x################ in VSIPToPFIoctl (cookie=<optimized out>, cmd=<optimized out>, data=<optimized out>, dataLen=<optimized out>, result=0x################) at datapath/esx/modules/vsip/vsip_pf/pf_vmk/msg2pf.c:9911
#3  0x################ in VSIPFlowGetRecordsForFilter (sol=<optimized out>, filter=filter@entry=0x################, data=data@entry=0x################) at datapath/esx/modules/vsip/vsip_flow.c:1684
#4  0x################ in VSIPDVFConfigFilterByName (fpAgentName=fpAgentName@entry=0x################ "vmware-sfw", filterName=filterName@entry=0x################ "nic-########-eth0-vmware-sfw.2", iter=iter@entry=0x################ <VSIPFlowGetRecordsForFilter>, 
    data=data@entry=0x################) at datapath/esx/modules/vsip/vsip_dvfilter.c:6575
#5  0x################ in VSIPFlowGetRecords (fpAgentName=fpAgentName@entry=0x################ "vmware-sfw", dvsPort=dvsPort@entry=0x################, cmdId=<optimized out>, buffer=buffer@entry=0x################ "\004", bufLen=bufLen@entry=########, 
    activeInfo=activeInfo@entry=0x################, inactiveInfo=0x################, dropInfo=0x################, noFlush=0 '\000', rttReset=0 '\000', includeAttrs=1 '\001', recCrit=0x################) at datapath/esx/modules/vsip/vsip_flow.c:2463
#6  0x################ in VSIPIoctlFlowData (cmd=<optimized out>, iocData=0x################, fnData=<optimized out>, result=<optimized out>) at datapath/esx/modules/vsip/vsip_fw_ioctl.c:1404
#7  0x################ in VSIPIoctlImpl (cmd=cmd@entry=22, req=req@entry=0x################, result=result@entry=0x################) at datapath/esx/modules/vsip/vsip_ioctl.c:148
#8  0x################ in VSIPCharDevIoctl (attr=<optimized out>, cmd=22, userData=############, callerSize=<optimized out>, result=0x################) at datapath/esx/modules/vsip/vsip_dev.c:55
#9  0x################ in VMKAPICharDevIoctl (handle=0x################, handle=0x################, ioctlResult=0x################, userData=############, cmd=22) at bora/vmkernel/main/vmkapi_char.c:649
#10 VMKAPICharDevDevfsWrapIoctl (handle=0x################, cmd=22, userData=############, ioctlResult=0x################) at bora/vmkernel/main/vmkapi_char.c:1507

 

Environment

VMware vDefend Firewall

Cause

A timing/race condition exists between the VSIP PF Purge Thread expiring/deleting a state object and the GETFLOWS ioctl thread (nsx-exporter) retrieving the same state object. Because memory associated with alg_state or alg_port is freed before state removal completes, a Use-After-Free (UAF) access triggers a host PSOD.

Resolution

This issue will be resolved in upcoming NSX releases.

Workaround: Disable or eliminate Application Level Gateway (ALG) flows if possible. Without active ALG flows, the specific multi-thread race condition will not trigger a host PSOD.