Error: "Server is overloaded" in NSX-T Manager due to excessive API calls from Aria Operations NSX-T adapter

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

The NSX-T Manager UI fails to load with error "Server is overloaded".

Review the following log file on the cloud proxy or data node for confirmation:
/storage/vcops/log/NSXTAdapter/NSXTAdapter_.log

You will see the following error:  

ERROR [NSX-T Metric Collector thread, <date and time> com.vmware.adapter3.nsxt.util.NsxtApiUtil.getTransportNodeInterfaceStatistics - Node Interface Statistics properties not retrieved due to 429 error, retrying the API.

Or view the logs from the UI if you do not have access to the appliances:

1. Go to Administration → Control Panel → Support Logs
2. Expand the collector node → COLLECTOR → adapters → NSXTAdapter
3. You will see a list of logs identified by NSXTAdapter_<internal ID>.log.  The newest log will not have a numerical value at the end.

Note: To find the correct logs, first check the Integrations page to verify which collector the adapter instance is currently assigned to.

Environment

• NSX-T 3.2.4, 4.1.x, 4.2.x
• Aria Operations 8.18.x

Cause

Excessive API calls from NSX-T adapter instance.

The NSX-T adapter instance is generating excessive API retries during the collection cycle, overwhelming the NSX-T management plane.

Resolution

Lower the retry attempt from NSX-T adapter instance:

1. Identify the collector for the NSX-T adapter instance:
   1. Navigate to Administration > Integrations.
   2. Locate the NSX-T adapter instance, click the vertical ellipsis(three dots), and select Edit
   3. Note the Collector / Group will identify which cloud proxy, data node, or collector group for the NSX-T adapter instance
2. Log into the identified cloud proxy or data node as root.
3. Edit the configuration file:  

   vi /usr/lib/vmware-vcops/user/plugins/inbound/NSXTAdapter3/conf/NSXT.properties

4. Locate the following parameter and change the value from 5 to 1:  

   API_RETRY_COUNT=1

5. Save the file by hitting "Esc" and then typing:

   :wq

6. The next collection cycle will show only 1 retry attempt.