Increased API Request Rates in VCF 9.x

Products

VMware NSX VCF Operations VCF Automation

Issue/Introduction

After upgrading to VMware Cloud Foundation (VCF) 9, users may observe a significant and sustained increase in API request trends on NSX Manager nodes compared to previous versions.

High baseline API traffic in their environments.
Syslog warnings indicating API rate limits are being approached or exceeded.
HTTP 429 Errors (Too Many Requests) appearing in Domain Manager logs.
Warnings triggered when API utilization exceeds 80% of the default limit.

Environment

VMware NSX
VCF Operations
VCF Automation

Cause

Architectural Changes in VCF 9

The increased API load is expected behavior resulting from the deeper integration and centralized management model of the VCF 9 stack.

1. Centralized Fleet Management (VCF Operations)
VCF 9 moves away from standalone component management. VCF Operations now continuously polls NSX Managers to maintain a "single pane of glass." This background traffic includes:

Unified Tag Management: Constant polling to enforce configuration consistency.

Centralized Security: Data aggregation for the unified security dashboard.

Compliance Syncing: Frequent checks for certificate and password management.

2. NSX VPC & VCF Automation Integration
VCF 9 introduces native NSX Virtual Private Clouds (VPCs). To support self-service provisioning, VCF Automation (formerly Aria Automation) uses high-frequency API-driven orchestration to sync state and manage dynamic network resources.

3. VCF Operations for Networks (vRNI)
When integrated, this component is designed to utilize up to 20% of total NSX API bandwidth (averaging ~20 calls per second, per node) to handle flow monitoring, metrics collection, and application discovery.

4. Post-Upgrade Convergence
The process of converging or importing workload domains during a VCF 9 upgrade triggers massive API spikes as transport nodes are created and synchronized across the stack.

Resolution

The default NSX Manager API rate limit is 100 requests per second. If these rates lead to timeouts or failures in automated tasks, use the following steps to adjust the environment:

A. Adjusting API Rate Limits

If your environment consistently hits the 100 req/sec , you can increase the threshold via the NSX Manager CLI:

Log into the NSX Manager via SSH using admin credentials.
To increase the limit (e.g., to 150 requests per second): set service http client-api-rate-limit 150
To disable the limit (recommended only temporarily during heavy migration/convergence tasks): set service http client-api-rate-limit 0

[!CAUTION]
Increasing API limits will result in higher CPU and Memory consumption on the NSX Manager nodes. Monitor resource utilization closely after making changes.

In the context of VCF 9, these limits become critical due to the increased polling from VCF Operations and Automation. Below is the technical summary of the documented limits and how to manage them.

1. Types of API Limits

The NSX API service enforces three distinct layers of protection to ensure management plane stability:

Limit Type	Default Value	Behavior on Violation
Per-Client Rate Limit	100 req/sec	Returns HTTP 429 (Too Many Requests).
Per-Client Concurrency	40 requests	Returns HTTP 503 (Service Unavailable).
Global Concurrency	199 requests	Returns HTTP 503 (Total processes on server).