"Delta Sync" between Local Managers and Global Managers is not working causing NSX Objects to be out of sync

Products

• VMware NSX

Issue/Introduction

• Error in the UI:

  data synchronization between sites failed for the FlowIdentifier(role='policy', namespace='LM_2_GM_NOTIFICATION')

• /etc/init.d/async-replicator-service status shows the async_replicator service is running. 
• The NSX local managers UI the Replication shows in a healthy state.
• In the /var/log/async-replicator/ar.log we see error similar to:
  

  WARN EventReportProcessor-1-3 EventReportSyslogSender 77561 MONITORING [nsx@6876 comp="global-manager" entId="ID" eventFeatureName="federation" eventSev="warning" eventState="On" eventType="queue_occupancy_threshold_exceeded" level="WARNING" subcomp="async-replicator"] Queue 

• In the /var/log/syslog on the GM we see the below:

  NSX 4298 MONITORING [nsx@6876 alarmId="########-####-####-####-########" alarmState="OPEN" comp="global-manager"
  
  entId="########-####-####-####-########" errorCode="MP701099" eventFeatureName="federation" eventSev="HIGH" eventState="On" eventType="gm_to_lm_synchronization_error" level="ERROR" nodeId="########-####-####-####-########" subcomp="monitoring"] Data synchronization between sites <site-name> (########-####-####-####-########) and <site-name> (########-####-####-####-########) failed for the FlowIdentifier{role='Policy', nameSpace='LM_2_GM_NOTIFICATION'} for an extended period. Reason: Out of sync since 1786628319016.

• In the /var/log/gmanager/gmanager.log:

  ERROR ClientReceiverFlowHandler-0 ClientDataReceiver 6487 - [nsx@6876 comp="global-manager" errorCode="MP1" level="ERROR" subcomp="global-manager"] Was not able to process received delta messages. Flow = FlowIdentifier{role='Policy', nameSpace='LM_2_GM_NOTIFICATION'}, site = ########-####-####-####-########.com.vmware.nsx.management.common.exceptions.PrecommitTransactionInterceptorException: Cannot invoke "java.util.Map.containsKey(Object)" because the return value of "java.lang.ThreadLocal.get()" is nullCaused by: java.lang.NullPointerException: Cannot invoke "java.util.Map.containsKey(Object)" because the return value of "java.lang.ThreadLocal.get()" is null

Environment

VMware NSX

Resolution

Rebooting the local mangers fixed the issue.