Edge datapath crashes and creates a core while performing operations on global address set table

Products

• VMware vDefend Firewall

Issue/Introduction

Impact : The issue causes a data path process crash, resulting in a core dump during packet processing within the address set match. This triggers an automatic restart of the data path and can lead to a High Availability (HA) failover.

Symptoms:

•   Edge Datapath Core Dumps: The system generates core files on the Edge node during operation.
•   Service Restarts: The datapath process restarts automatically following a crash.
•   HA Failover:Traffic may shift to the standby Edge node due to the active node's datapath failure.
•   Specific Backtrace Keywords: A generated core dump on the Edge is a match if the backtrace contains the following keyword:  pfr_match_addr_ke pf_rule_exact_match pf_fastpath_exact_l4_match pf_fastpath_match_l7 

If the generated core dump on edge have following keyword in backtrace then it is a match

#0  pfr_match_addr_ke (kt=0x*****, kt@entry=0x************, a=0x************, af=* '\***') at datapath/pf/pf/pf_table.c:****
#1  0x************ in pfr_match_addr (af=<optimized out>, a=<optimized out>, kt=0x************) at datapath/pf/pf/net/pfvar.h:****
#2  pf_rule_exact_match (kif=0x************, r=0x************, pd=0x************, sport=<optimized out>, dport=<optimized out>, direction=<optimized out>, flags=*, mismatch=0x************, cache=0x*)
    at datapath/pf/pf/pf.c:****
#3  0x************ in pf_fastpath_exact_l4_match (kif=kif@entry=0x************, rlist=rlist@entry=0x************, pd=pd@entry=0x************, sport=sport@entry=*****, dport=dport@entry=*****, 
    direction=direction@entry=*, cache=0x*) at datapath/pf/pf/pf.c:****
#4  0x************ in pf_fastpath_match2_l7 (kif=kif@entry=0x************, ruleset=ruleset@entry=0x*, pd=pd@entry=0x************, sport=<optimized out>, dport=<optimized out>, direction=<optimized out>, 
    error=<optimized out>, mask=<optimized out>, rs_num=<optimized out>, service_rules=<optimized out>, no_fw=<optimized out>, rlookup=<optimized out>, prlist=<optimized out>, curr_attr_state=<optimized out>, 
    next_attr_state=<optimized out>, attribute_log=<optimized out>, tv_now=<optimized out>, ac=<optimized out>, url_fil=<optimized out>, l7ap_uuid=<optimized out>, tls_params=<optimized out>, 
    need_log=<optimized out>, local_ctx=0x*, rs_lb=*) at datapath/pf/pf/pf.c:****
#5  0x************ in pf_fastpath_match_l7 (kif=kif@entry=0x************, ruleset=0x*, ruleset@entry=0x************, pd=pd@entry=0x************, sport=sport@entry=*****, dport=dport@entry=*****, 
    direction=direction@entry=*, error=<optimized out>, mask=<optimized out>, rs_num=<optimized out>, service_rules=<optimized out>, no_fw=<optimized out>, rlookup=<optimized out>, rlist=<optimized out>, 
    curr_attr_state=<optimized out>, next_attr_state=<optimized out>, attribute_log=<optimized out>, need_log=<optimized out>, tv_now=<optimized out>, ac=<optimized out>, url_fil=<optimized out>, 
    l7ap_uuid=<optimized out>, tls_params=0x************, local_ctx=0x*, rs_lb=*) at datapath/pf/pf/pf.c:****
#6  0x************ in pf_rule_match_l7 (local_ctx=0x*, tls_params=0x************, l7ap_uuid=0x************ "", url_fil=0x************, ac=0x*, tv_now=0x************, need_log=<optimized out>, 
    attribute_log=0x************, next_attr_state=0x************, curr_attr_state=0x************, rlist=0x************, rlookup=* '\***', no_fw=<optimized out>, service_rules=<optimized out>, 
    mask=0x************, error=0x************, matched=<synthetic pointer>, tag=0x************, dport=<optimized out>, sport=<optimized out>, pd=0x************, direction=*, m=0x************, 
    r=<optimized out>, ruleset=0x************, kif=0x************) at datapath/pf/pf/pf.c:*****
#7  pf_test_tcp (rm=rm@entry=0x************, sm=sm@entry=0x************, state=state@entry=0x************, direction=<optimized out>, kif=kif@entry=0x************, m=m@entry=0x************, 
    off=<optimized out>, h=<optimized out>, rlookup=<optimized out>, next_attr_state=<optimized out>, ac=<optimized out>, pd=<optimized out>, ethtype=<optimized out>, am=<optimized out>, rsm=<optimized out>, 
    ifq=<optimized out>, inp=<optimized out>, reason=<optimized out>) at datapath/pf/pf/pf.c:*****
#8  0x************ in pf_validate_state (kif=kif@entry=0x************, state=state@entry=0x************, rule=rule@entry=0x************, anchor_rule=anchor_rule@entry=0x************, 
    orig_pd=orig_pd@entry=0x************, ethtype=<optimized out>, paction=0x************, next_attr_state=0x************, sync_ac=<optimized out>, intf_change=* '\***') at datapath/pf/pf/pf.c:*****
#9  0x************ in pf_validate_session (kif=kif@entry=0x************, state=0x************, pd=pd@entry=0x************, ethtype=ethtype@entry=****) at datapath/pf/pf/pf.c:*****
#10 0x************ in pf_test_state_tcp (state=state@entry=0x************, direction=direction@entry=*, kif=kif@entry=0x************, m=m@entry=0x************, off=<optimized out>, off@entry=**, 
    h=h@entry=0x************, pd=0x************, ethtype=****, reason=0x************, check_only=*, drop_rst=0x************) at datapath/pf/pf/pf.c:*****
#11 0x************ in pf_test (dir=dir@entry=*, ifp=ifp@entry=0x************, m0=m0@entry=0x************, eh=eh@entry=0x*, ethtype=ethtype@entry=****, inp=inp@entry=0x*, metadata=0x************, 
    check_only=*, pfmi=0x************) at datapath/pf/pf/pf.c:*****
#12 0x************ in dpdk_pf_test (dir=dir@entry=*, iface_type=iface_type@entry=* '\***', pkt=<optimized out>, pkt@entry=0x************, eth_hdr_len=eth_hdr_len@entry=**, 
    cookie=cookie@entry=0x************, eth=eth@entry=0x*, ether_type=<optimized out>, meta=<optimized out>) at datapath/pf/pf_glue/glue.c:****
#13 0x************ in firewall_process_packet (vrf_id=vrf_id@entry=*, m=m@entry=0x************, pkt_type=pkt_type@entry=FIREWALL_PKTTYPE_BRIDGE, hook=hook@entry=FIREWALL_HOOK_PREROUTING, 
    p_output=p_output@entry=0x************) at datapath/firewall.c:****
#14 0x************ in lswitch_firewall_process_packet (m=<optimized out>, ingress_lp=ingress_lp@entry=0x************) at datapath/lswitch.c:****
#15 0x************ in lswitch_input_from_lport (m=<optimized out>, m@entry=0x************, ingress_lp=0x************) at datapath/lswitch.c:****
#16 0x************ in iface_outputx (m=m@entry=0x************, ifp=0x************, if_port=if_port@entry=*** '\***', if_type=<optimized out>, is_repl=is_repl@entry=false, span_prepend_eth_type=*, 
    span_prepend_eth=false, rl=0x************ <rl>, v=0x************ <VLM_lswitch>) at datapath/iface-impl.h:****
#17 0x************ in iface_output (is_repl=false, if_type=<optimized out>, if_port=*** '\***', ifp=<optimized out>, m=0x************, rl=0x************ <rl>, v=0x************ <VLM_lswitch>)
    at datapath/iface-impl.h:****
#20 fpn_main_loop (unused=<optimized out>) at datapath/main-loop.c:****

Environment

• NSX-T 4.1.2.4
• NSX-T 4.2.3.2

Cause

A race condition occurs because two threads—the IPC thread and the purge thread—simultaneously access the global address set table. The IPC thread tries to insert an entry while the purge thread tries to remove one, leading to memory corruption and subsequent null or dangling pointer crashes.

Resolution

The issue will get fixed in future version