Onboarding or offboarding Local Manager to GM is failing on a federated setup. (Error code: 529013)

Products

• VMware NSX

Issue/Introduction

• While attempting to onboard or offboard an NSX Local Manager to or from Global Manager,  you receive the following error:

Error: Message size 59,767,760 is larger than supported size 51,200,000 in Global Manager transaction queue. Full sync to Site will be requested to bring Site up to speed with GM. If deleting site, please delete some resources before retrying. (Error code: 529013)

• In Global Manager logs /var/log/gmanager/gmanager-ui.log you will observe the following:

INFO http-nio-127.0.0.1-64440-exec-42 NsxBaseRestController 4069 SYSTEM [nsx@6876 comp="global-manager" level="INFO" subcomp="global-manager"] Error in API /gm/api/v1/infra/sites/{site-id} caused by exception com.vmware.nsx.management.common.exceptions.PrecommitTransactionInterceptorException:  {"moduleName":"Policy","errorCode":529013,"errorMessage":"Message size 59,767,964 is larger than supported size 51,200,000 in Global Manager transaction queue. Full sync to Site will be requested to bring Site up to speed with GM. If deleting site, please delete some resources before retrying.","details":"com.vmware.nsx.management.policy.policyframework.exceptions.InvalidFederationOperationException: Message size 59,767,964 is larger than supported size 51,200,000 in Global Manager transaction queue. Full sync to Site will be requested to bring Site up to speed with GM. If deleting site, please delete some resources before retrying."}

INFO http-nio-127.0.0.1-64440-exec-42 NsxBaseRestController 4069 SYSTEM [nsx@6876 audit="true" comp="global-manager" level="INFO" subcomp="global-manager"] UserName:'admin' ModuleName:'Policy' Operation:'PUT@/api/v1/infra/sites/{site-id}' Operation status: 'failure' Error: Message size 59,767,964 is larger than supported size 51,200,000 in Global Manager transaction queue. Full sync to Site will be requested to bring Site up to speed with GM. If deleting site, please delete some resources before retrying.

ERROR pool-122-thread-1 GmFacadeInterceptorValidator 4069 POLICY [nsx@6876 comp="global-manager" errorCode="PM500518" level="ERROR" reqId="#######-####-####-####-########" subcomp="global-manager" username="system"] The API readTraceGlobalConfig is invalid on the global manager.

• If offboarding, you might see an error in the Global Manager: System > Location Manager: 
  Something went wrong. We are unable to show the information about the Global Manager or Locations.
• If offboarding, you might see a reference to the old Local Manager site and an error in the Global Manager: System > System Overview:
  Error: General error has occurred. (Error code: 100)

Environment

VMware NSX-T 3.2.3.1

Cause

The system is trying to push more than 10k objects, the payload size is going beyond 51,200,000 bytes limit and hence the error. 

Resolution

This issue is resolved in VMware NSX-T 3.2.4 and 4.1.3, available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.