Connection dropped due to iptables rate limit on NSX-T Manager

Products

• VMware NSX

Issue/Introduction

• Errors in the dmesg log indicating dropped connections due to rate limits.
  file path in logs :  system/dmesg

  dmesg:
  
  [355518.907296] Dropped per conn limit: IN=eth0 OUT= MAC=mac1 SRC=192.###.###.1 DST=192.###.###.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=49906 DF PROTO=TCP SPT=49611 DPT=1234 WINDOW=64240 RES=0x00 SYN URGP=0
  
  [355519.938940] Dropped per conn limit: IN=eth0 OUT= MAC=mac2 SRC=192.###.###.1 DST=192.###.###.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=49907 DF PROTO=TCP SPT=49611 DPT=1234 WINDOW=64240 RES=0x00 SYN URGP=0

• This can be verified live by running the following command on the NSX Manager CLI on root mode:

dmesg -T

Sample Output: [Timestamp] [efw-udp-ext] IN= OUT=eth0 SRC=##.##.##.## DST=##.##.##.## LEN=67 TOS=0x00 PREC=0x00 TTL=64 ID=65473 DF PROTO=UDP SPT=45708 DPT=53 LEN=47 UID=991 GID=991

Impact/Risks:

• NSX controller connection towards the Transport node might get impacted.
• NSX-T adapters cannot register to the NSX manager.
• Antrea-NSX integration fails to function as expected, hindering network functionality in the Tanzu environment.

Environment

VMware NSX

Cause

The issue is caused by iptables rate limiting on the NSX Manager. When a source IP (such as a gateway ,interworking pod or vulnerability scanner) exceeds the configured connection threshold (defaulting to 10/s in some versions).

Resolution

Resolution

Identify the source IP attempting constant connections to the NSX Manager.

 Check NSX manager- Login to NSX manager, and run dmesg -T, if there is following errors/warnings.

[Tue May dd 10:09:57 20yy] IPTables-Dropped: IN=eth0 OUT= MAC=00:50:56:##:##:##:00:50:56:##:##:##:##:## SRC=10.###.###.1 DST=10.###.###.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=41119 DF PROTO=TCP SPT=30439 DPT=1235 WINDOW=64240 RES=0x00 SYN URGP=0
[Tue May dd 10:09:59 20yy] IPTables-Dropped: IN=eth0 OUT= MAC=00:50:56:##:##:##:00:50:56:##:##:##:##:## SRC=10.###.###.1 DST=10.###.###.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=41120 DF PROTO=TCP SPT=30439 DPT=1235 WINDOW=64240 RES=0x00 SYN URGP=0
[Tue May dd 10:10:04 20yy] IPTables-Dropped: IN=eth0 OUT= MAC=00:50:56:##:##:##:00:50:56:##:##:##:##:## SRC=10.###.###.1 DST=10.###.###.2 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=41121 DF PROTO=TCP SPT=30439 DPT=1235 WINDOW=64240 RES=0x00 SYN URGP=0

 

1. External Scanners

If the source is a vulnerability scanner or non-critical monitoring tool, it is strongly recommended to reduce the scan frequency and extend the intervals between probes.

2. Valid Source (e.g., Tanzu/Antrea Clusters)

If the source is valid and requires higher rate limit, follow these steps For detailed step-by-step procedures, refer to 317179