40 Requests Per Second observed with 20 RPS limit - VMware NSX Data Sources

Products

• VCF Operations for Networks

Issue/Introduction

• Each NSX Manager node receives 40 requests per second from a Collector VM.
  
  For example:
  $ cut -d ' ' -f 1,2 var/log/proxy/envoy_access_log*|grep <AON_COLLECTOR_IP>| sed -E 's/\.[0-9]{3}Z//'|sort|uniq -c|sort -nr|head -n1
40 [<DATE>T01:01:23] <AON_COLLECTOR_IP>
  
  
• A period of inactivity before 40 requests per second.
  
  For example if the timestamp is <DATE>T01:01:23, you will see idleness in <DATE>T01:01:22.
  

  $ cut -d ' ' -f 1,2 var/log/proxy/envoy_access_log*|grep <AON_COLLECTOR_IP>| sed -E 's/\.[0-9]{3}Z//'|sort|grep <DATE>T01:01:23|wc -l
40

  $ cut -d ' ' -f 1,2 var/log/proxy/envoy_access_log*|grep <AON_COLLECTOR_IP>| sed -E 's/\.[0-9]{3}Z//'|sort|grep <DATE>T01:01:22|wc -l
0

• VCF Operations for Networks is configured with a 20 requests per second by default.
  Reducing API calls sent from VCF Operations for Networks to NSX Managers

Environment

VCF Operations for Networks

VMware NSX Manager

Cause

This is expected behavior and is a result of the "bursty" rate-limiting logic implemented within the Collector for VMware NSX.

The system is set to 20 requests per second by default. During periods of idleness or inactivity, the RateLimiter stores "permits." It can store up to 20 permits during these quiet periods. When a sudden spike in traffic occurs after an idle period, the RateLimiter immediately consumes the 20 stored permits with zero wait time. Simultaneously, it continues to permit the standard 20 requests for that current second.

20 stored permits (burst) + 20 standard permits (current rate) = 40 total requests allowed in that initial second.

Resolution

No resolution or configuration change is required. The observation of 40 requests per second is a validated functional behavior of the RateLimiter implemented in a Collector for NSX Data Sources intended to accommodate bursty traffic patterns.