Reducing API calls sent from VCF Operations for Networks to NSX Managers

Products

VCF Operations for Networks VMware NSX

Issue/Introduction

Potential Issues with NSX Managers:

While working with NSX Managers, you may encounter the following issues:

API Rate Limiting: When making API calls against the NSX Managers, you might receive the following response:
```
Response code '429 - Too Many Requests'.
```
This indicates that the system has exceeded its maximum number of allowable API requests within a defined period.
Log Warnings: Upon reviewing the NSX Manager /var/log/syslog, you may notice a warning message similar to the following:
```
"A new maximum of client API rate has exceeded 80% of the allowed limit. Max = 111, client = <NSX-T account credentials>, limit = 100".
```
This log entry highlights that the client has reached or exceeded 80% of the allowed API rate limit, indicating a potential risk of hitting the maximum threshold.

Symptoms may present in a way that may suggest your VCF Operations for Networks nodes are going down.

NOTE: VCF Operations for Networks was formerly named Aria Operations for Networks (AON), and prior to that was named vRealize Network Insight (vRNI).

Environment

VCF Operations for Networks

Resolution

In scenarios where the NSX Manager API rate limit is exceeded, it is typically due to an excessive number of API requests being made within a short time frame.

By default, NSX Manager supports up to 100 API calls per second for each manager node.

When NSX is configured as a data source within VCF Operations for Networks, the system is optimized to make a maximum of 20 API calls per second per NSX Manager node. This ensures that VCF Operations for Networks consumes only 20% of the total NSX API rate bandwidth, allowing the system to operate efficiently without overloading the manager node's API capacity.

Important Considerations:

It is critical to note that VCF Operations for Networks is specifically configured to utilize no more than 20% of the total NSX API rate bandwidth by making 20 API calls per second per NSX manager node.
20 API calls per second per NSX manager node is the optimal rate for VCF Operations for Networks to function efficiently and maintain key features such as below mentioned:

Flow Monitoring
Application Discovery
Metrics Collection
Reducing the API rate from VCF Operations for Networks will impact above features, causing delays in data collection, which could affect the performance and accuracy of these functionalities.
VCF Operations for Networks support recommends making any rate adjustments carefully, considering the trade-offs in system performance.
By maintaining the default configuration (20 API Calls), you ensure that VCF Operations for Networks operates within the ideal API call threshold for NSX, balancing performance and API bandwidth efficiently.
Keep in mind that this should only be done if absolutely necessary, as it may impact the system's functionality.

If you need to reduce the number of API calls from VCF Operations for Networks for any reason, this can be done via VCF Operations for Networks database.

Open a support case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.

Additional Information

INVESTIGATIVE APPROACH:

Analyze the IP addresses in the /var/log/envoy_access_log events in NSX Manager logs that contain the string " 429 " with a command such as (entered from directory path var/log/proxy):
- grep " 429 " envoy_access_lo* | sed "s#txt:# #"g | cut -d ' ' -f3 | sort | uniq -c
The output of this might look something like this:
- 56 ##.##.##.##
  415 ##.##.##.###
  89 ##.##.###.###
  1198 ##.###.###.###
- Where:
  - there are 4 different unique IP addresses (in this example, ##.##.##.##, ##.##.##.###, ##.##.###.### and ##.###.###.###)
  - The numbers 56, 415, 89 and 1198 represent the frequency count of the " 429 " (too many requests) events in the logs in that directory
You may find that one or more of the IP addresses indicated belongs to a VCF Operations for Networks Platform or Collector node in your environment.
If so, you have the option of either
1. reducing the number of API requests per second coming from the VCF Operations for Networks Platform or Collector node; or
2. increasing the allowed threshold limit in the NSX Manager to a level that is higher than the "Limit = " value (which is 100 by default)
To determine the magnitudes of the violations of the threshold, from the /var/log/ directory of the NSX Manager, you can run a command like:
- grep "New maximum of client API rate has exceeded 80% of the allowed limit." syslo* | grep -v "message repeated" | cut -d ' ' -f23-25 | sed "s#,##"g | sort -n -r -k3 | uniq -c | head
The output of this might look something like this:
- 2 Max = 136
  1 Max = 135
  3 Max = 134
  6 Max = 133
  5 Max = 132
  4 Max = 131
  8 Max = 130
  13 Max = 129
  18 Max = 128
  12 Max = 127
In this example, you could see that the highest value seen in the logs was 136, so using the option in 4b. above, and choosing a value such as "180" would be more than sufficient to accommodate the higher number of requests per second. (80% of 180 = 144 which is higher than 136).

To increase the threshold limit on the NSX Manager, please see KB 427427 - NSX UI Error: Client 'admin' exceeded request rate of 100 per second (Error code: 102)