Edge stops forwarding traffic and Gateway failover occurs

Products

• VMware NSX

Issue/Introduction

• NSX-T Data Center 3.1.3.8, 3.2.0, 3.2.1
• Edge stops forwarding traffic and Gateway failover occurs
• Edge admin cli may become unresponsive
• Edge syslog may have logs similar to this example

2023-09-25T14:56:22.373Z Edge01 NSX 8478 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu1" level="WARN"] blocked 2048000 ms waiting for main to quiesce

 

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 4.x
VMware NSX-T Data Center 3.x

Cause

This issue occurs when datapath thread gets stuck in a deadlock state due to a race condition. This issue is triggered when a large amount of fragmented traffic hits multiple Edge cores.

Resolution

This issue has is resolved in NSX 3.2.2 and 4.1.0.

Workaround:

To prevent the issue from occurring, disable all Gateway Firewalls on the Edge

    Security -> Gateway Firewall -> Actions -> General Settings -> Gateway Firewall Status

If an Edge is already in this state, restart the dataplane as admin user to resolve the deadlock issue

  #restart service dataplane