Aria Automation is causing NSX-T to alert its exceeding its Client API Limits

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• The Aria Automation provisioning service logs located under /var/log/services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log have '429 Too Many Requests' messages similar to:

INFO provisioning [host='provisioning-service-app-<id>' thread='OkHttp https://<NSX-T_FQDN>/api/v1/fabric/compute-collections?node_id=<nodeid>' user='' org='' trace='' parent='' span=''] c.v.m.a.n.NsxTRetryRequestInterceptor.log:454 - RetryRequestInterceptor: Response code '429 Too Many Requests'. Retrying
 

• An alert is generated in Log Insight for "NSX-T API limit above threshold":

New maximum of client API rate has exceeded 80% of the allowed limit. Max = 111, client = <NSX-T cloud account credentials>, limit = 100

Environment

VMware Aria Automation 8.x

Cause

The NSX-T enumeration process requires Aria Automation to make a significant amount of api calls to the NSX-T managers.

Resolution

The issue is pending a resolution. NSX-T version 3.2 & higher provides a more efficient API but in order to maintain compatibility with older existing NSX-T versions it cannot yet be leveraged by Aria Automation.

Workaround:

If the issue impacts functionality for example with NSX-T data-collections intermittently failing in Aria Automation then an external loadbalancer can be configured in front of the NSX-T manager appliances to balance the requests across the NSX manager cluster.

The process of deploying and configuring a loadbalancer in front of the NSX managers is outlined in the documentation.
 

Alternatively the api limits can be increased in NSX by following the procedure outlined in the NSX documentation. However this is not the preferred approach due to the potential for performance overhead and overuse by other client APIs.

Additional Information

Impact/Risks:
There is a retry mechanism built in to the Aria Automation NSX-T enumeration code to retry any failed api request so the issue is typically only cosmetic in nature. NSX-T cloud account data-collections occur automatically every 10 minutes.