Aria Automation is causing NSX-T to alert its exceeding its Client API Limits

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• The Aria Automation provisioning service logs located under /var/log/services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log have '429 Too Many Requests' messages similar to:

INFO provisioning [host='provisioning-service-app-<id>' thread='OkHttp https://<NSX-T_FQDN>/api/v1/fabric/compute-collections?node_id=<nodeid>' user='' org='' trace='' parent='' span=''] c.v.m.a.n.NsxTRetryRequestInterceptor.log:454 - RetryRequestInterceptor: Response code '429 Too Many Requests'. Retrying

• An alert is generated in Log Insight for "NSX-T API limit above threshold":

New maximum of client API rate has exceeded 80% of the allowed limit. Max = 111, client = <NSX-T cloud account credentials>, limit = 100

Environment

Aria Automation 8.x

NSX-T 3.2 or later versions.

Cause

The NSX-T enumeration process requires Aria Automation to make a significant amount of API calls to the NSX-T manager nodes.

Resolution

The issue is pending a resolution. NSX-T version 3.2 & higher provides a more efficient API but in order to maintain compatibility with older existing NSX-T versions it cannot yet be leveraged by Aria Automation.

Workaround:

 

If the issue impacts functionality for example with NSX-T data-collections intermittently failing in Aria Automation then an external load balancer can be configured in front of the NSX-T manager appliances to balance the requests across the NSX manager cluster.

The process of deploying and configuring a load balancer in front of the NSX managers is outlined in: Provisioning a Load Balancer for the VMware NSX Management Cluster.

 

Alternatively, the client API limits can be increased in NSX by following the procedure outlined in NSX-T Data Center Global Manager REST API reference. However this is NOT the preferred approach due to the potential for performance overhead and overuse by other client APIs.

Additional Information

Impact/Risks:

There is a retry mechanism built in to the Aria Automation NSX-T enumeration code to retry any failed api request so the issue is typically only cosmetic in nature. NSX-T cloud account data-collections occur automatically every 10 minutes.