NSX upgrade fails on SDDC - Client 'admin' exceeded request rate of 100 per second

Products

• VMware Cloud Foundation

Issue/Introduction

NSX-T upgrade fails for parallel cluster upgrade on a VCF environment on "NSX-T upgrade postcheck stage"

Environment

• VMware Cloud Foundation 3.x
• VMware Cloud Foundation 4.x
• VMware Cloud foundation  5.x

Cause

NSX-T supports default API request as 100 request per second. Since it is a parallel cluster upgrade it might exceed the limit which eventually fails. It points at the user "admin" since that is the user for API 

lcm-debug.log

ERROR [vcf_lcm,############38ac,de34,upgradeId=########-####-####-####-#######2bc4,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=######-#####-#####-###-##:_ParallelClusterUpgradeElement,bundleElementId=########-####-####-####-########396c] [c.v.e.s.l.p.i.n.s.NsxtHostClusterParallelUpgradeStageRunner,Async-10] Caught exception {"messages":[],"data":{"strValue":{"fields":{"error_message":{"value":"Client \#####user_admin\u0027 exceeded request rate of 100 per second"},"error_cod
e":{"value":102},"module_name":{"value":"common-services"}},"name":"struct"}},"errorType":{"name":"SERVICE_UNAVAILABLE"},"stackTrace":[],"suppressedExceptions":[]} while performing stage NSX_T_HOST_TYPE_POSTCHECK_STAGE. Returning status NSXT_HOST_CLUSTER
_UPGRADE_FAILED_POSTCHECK
ERROR [vcf_lcm,############38ac,de34,upgradeId=########-####-####-####-#######2bc4,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=######-#####-#####-###-##:_ParallelClusterUpgradeElement,bundleElementId=########-####-####-####-########396c] [c.v.e.s.l.p.i.n.s.NsxtHostClusterParallelUpgradeStageRunner,Async-10] Exception :
com.vmware.vapi.std.errors.ServiceUnavailable: ServiceUnavailable (com.vmware.vapi.std.errors.service_unavailable) => {
    messages = [],
    data = struct => {error_message=Client 'admin' exceeded request rate of 100 per second, error_code=102, module_name=common-services},
    errorType = SERVICE_UNAVAILABLE
}
        at com.vmware.vapi.std.errors.ServiceUnavailable._newInstance2(ServiceUnavailable.java:187)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Resolution

Workaround:

1. Get a ssh session to the VIP assigned NSX-T manager 
2. Login as admin
3. Run the following command 
   get service http 
   

   Similar Output

   Service name:	http
   Service state:	running
   Session timeout:	1800 seconds
   Connection timeout:	30 seconds
   Client API rate limit:	100 requests/sec
   Client API concurrency limit:	40 connections
   Global API concurrency limit:	199 connections
   Redirect host:	(not configured)

4. Run the following command to set the request to 120
   
   set service http client-api-rate-limit 120
5. Retry the upgrade from SDDC manager 

We recommend setting the client-api-rate-limit back to 100 once the upgrade has completed.