NSX-T Edge node datapath service may crash when two VRF's on the same edge node have REJECT firewall rule

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• VTEP tunnels on the NSX-T edge node may be unavailable.
• IPSEC VPN session(s) may be seen in an UNKNOWN state and are not working.
• On the NSX-T edge node the datapath service may crash and generate core dumps, as can be seen in the NSX-T edge node syslog:

<180>1 2021-03-19T09:18:17.263Z nsx-edge-01 NSX 28552 - [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="WARNING"] Core file generated: /var/log/core/core.dp-fp:0.1611047897.26431.0.11.gz

• You may also see the following error's in the NSX-T edge node syslog:

180>1 2021-01-19T16:04:29.283734+01:00 t-nsx-edge-01 edge-appctl - - - edge-appctl 732 unixctl SYSTEM[nsx@6876 comp="nsx-edge" subcomp="edge-appctl.unixctl" level="WARN"] failed to connect to /var/run/vmware/edge/dpd.ctl
<179>1 2021-01-19T15:04:29.284Z t-nsx-edge-01 NSX 1167 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="1869" level="ERROR" errorCode="MPA14007"] Return Code: 1
<179>1 2021-01-19T15:04:29.284Z t-nsx-edge-01 NSX 1167 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="1869" level="ERROR" errorCode="MPA14006"] Error Message Found: 2021-01-19T15:04:29Z edge-appctl 732 unixctl [WARN] failed to connect to /var/run/vmware/edge/dpd.ctl #012edge-appctl: cannot connect to "/var/run/vmware/edge/dpd.ctl" (Protocol error)#012
<179>1 2021-01-19T15:04:29.284Z t-nsx-edge-01 NSX 1167 - [nsx@6876 comp="nsx-edge" subcomp="agg-service" tid="1869" level="ERROR" errorCode="MPA14006"] Unable to execute edge-appctl command on Edge

• You have two VRF logical routers deployed, attached to a T0 logical router, both VRF logical routers are on the same edge node and have REJECT firewall rules.

Environment

VMware NSX-T Data Center

Cause

This issue can occur if there are two VRF's on the same NSX-T edge node and the logical router has a REJECT rule. 
This can lead to a circular REJECT policy.
The issue occurs when a TCP reset packet attempts to return, but is stopped by the REJECT rule in the first VRF.

Resolution

This issue is resolved in VMware NSX-T 3.1.1, available at Broadcom downloads.
This issue is resolved in VMware NSX-T 3.0.3, available at Broadcom downloads.

Workaround:
If you are unable to upgrade at this time, you can convert the logical router REJECT rule to a DROP rule, to prevent the issue from occurring again.