NSX-T edge nodes state become "unknown"
search cancel

NSX-T edge nodes state become "unknown"

book

Article ID: 367385

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Edge nodes enter an unknown state, and the BGP, BFD sessions go down.

---

syslog.42.gz:2024-04-19T16:09:27.373Z xx-yyy-z01-en02.cloud.fire.net NSX 1180088 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu2" level="WARN"] blocked 4096000 ms waiting for dp-ipc31 to quiesce
syslog.41.gz:2024-04-19T16:10:08.077Z xx-yyy-z01-en02.cloud.fire.net NSX 1180088 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="dp-si-purge5" level="WARN"] blocked 4096000 ms waiting for dp-ipc31 to quiesce

---

6674:2024-04-22T12:01:12.350Z xx-yyy-z01-en02.cloud.fire.net NSX 1894731 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" level="ERROR"] pf_snat_port_delete_alarm_processing: failed to find snat hash entry. NAT addr: zoo373b, daddr: cat373b, dport:20022, vrf: 202, num of snat ips crossing threshold: 0
6675:2024-04-22T12:01:12.243Z xx-yyy-z01-en02.cloud.fire.net 111122223333 1893338 - -  2024-04-22T12:01:12Z datapathd 1894731 firewalldp [ERROR] pf_snat_port_delete_alarm_processing: failed to find snat hash entry. NAT addr: zoo373b, daddr: cat373b, dport:20022, vrf: 202, num of snat ips crossing threshold: 0
6683:2024-04-22T12:01:12.356Z xx-yyy-z01-en02.cloud.fire.net datapath-systemd-helper 1894616 - -  2024-04-22T12:01:12Z datapathd 1894731 firewalldp [ERROR] pf_snat_port_delete_alarm_processing: failed to find snat hash entry. NAT addr: zoo373b, daddr: cat373b, dport:20022, vrf: 202, num of snat ips crossing threshold: 0
7978:2024-04-22T12:01:13.609Z xx-yyy-z01-en02.cloud.fire.net NSX 1894731 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" tname="dp-fw-purge11" level="ERROR"] pf_snat_port_delete_alarm_processing: failed to find snat hash entry. NAT addr: dog373b, daddr: bat373b, dport:47873, vrf: 174, num of snat ips crossing threshold: 0

---

2024-04-26T00:29:33.834Z xx-yyy-z01-en01.cloud.fire.net NSX 1492808 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" level="ERROR"] pf_snat_port_add_alarm_processing: failed to allocate snat hash entry. NAT addr: czoo373b, daddr: key8d401, dport:39173, vrf: 101, num of snat ips crossing threshold: 0
2024-04-26T00:29:33.834Z xx-yyy-z01-en01.cloud.fire.net NSX 1492808 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" level="ERROR"] pf_snat_port_add_alarm_processing: failed to allocate snat hash entry. NAT addr: czoo373b, daddr: alt5f001, dport:39173, vrf: 101, num of snat ips crossing threshold: 0
2024-04-26T00:29:33.834Z xx-yyy-z01-en01.cloud.fire.net NSX 1492808 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewalldp" level="ERROR"] pf_snat_port_add_alarm_processing: failed to allocate snat hash entry. NAT addr: czoo373b, daddr: que7d31b, dport:48385, vrf: 101, num of snat ips crossing threshold: 0

 

Resolution

This is a known deadlock issue without any workaround.

The issue is fixed in the following versions

  • 4.1.2.2
  • 4.1.0.4
  • 3.2.4