GM To LM Synchronization Warning

Products

• VMware NSX

Issue/Introduction

• NSX Alarm "GM To LM Synchronization Warning" is present
• If the alarm is manually resolved, it retriggers again
• There are no sync issues present, objects created on the Global Managers are propagated to the Local Managers as expected
• On the Local Manager /var/log/async-replicator/ar.log, the following logging is seen

<DATE>T03:23:11.823Z  INFO nsx-rpc:unix:///var/run/vmware/nsx-opsagent/alarms-provider-service.sock:user-executor-0 EventSource 87147 MONITORING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="async-replicator"] EventSource: Sync triggered. featureName: federation, eventType: gm_to_lm_synchronization_warning, entityId: UUID, status: true, context: {"site_id":"SITE ID","remote_site_id":"UUID","site_name":"SITE_NAME","remote_site_name":"SITE_NAME","flow_identifier":"FlowIdentifier{role='Policy', nameSpace='LM_2_GM_ONBOARD_CONFIG'}","sync_issue_reason":"Remote site disconnected"}

• A leadership change has occurred for the ArAlarmService service, on the Local Manager /var/log/cbm/gmle-leadership-lease.log shows leaderId change 
  
  <DATE>T11:59:08.285Z  INFO ClusteringRpcServer-Leadership-Thread1 LeadershipRpcHandler 84807 - [nsx@6876 comp="global-manager" level="INFO" s2comp="leadership-rpc-handler" subcomp="cbm"] Renewing the leadership lease of group <GROUP_ID>, new lease LeadershipLease{serviceName=ArAlarmService, leaderId=<MANAGER_UUID_1>, leaseVersion=1601848, revocationCount=0, serviceWeight=1, serviceWeightCategory=SMALL, leaseId=<LEASE_ID>, relinquishInProgress=false}

<DATE>T12:00:41.312Z  INFO ClusteringRpcServer-Leadership-Thread1 LeadershipRpcHandler 84807 - [nsx@6876 comp="global-manager" level="INFO" s2comp="leadership-rpc-handler" subcomp="cbm"] Renewing the leadership lease of group <GROUP_ID>, new lease LeadershipLease{serviceName=ArAlarmService, leaderId=<MANAGER_UUID_2>, leaseVersion=1601856, revocationCount=0, serviceWeight=1, serviceWeightCategory=SMALL, leaseId=<LEASE_ID>, relinquishInProgress=false}

Environment

VMware NSX 4.x
VMware NSX-T 3.x

Cause

After a Synchronization alarm validly triggers, if the leader of the ArAlarmService changes from an NSX Manager to another NSX Manager before the alarm resolves then the alarm cannot be cleared.

Resolution

This issue is resolved in VMware NSX 4.2.0 available at Broadcom Downloads.

To resolve the alarm, restart the Async Replication service on the Local NSX Manager

1. ssh as admin user to any NSX Manager
2. Identify the leader of the Async Replicator service
   
   #get cluster config
   
   In the section ASYNC_REPLICATOR, check for the UUID of the Ar Alarm Service
    ArAlarmService      1       SMALL     <node UUID>       1641194
   
   
3. ssh to this node if not already on it and restart the service
   
   #systemctl stop async-replicator-service
#systemctl start async-replicator-service
   
   
4. Resolve the Synchronization Alarm on the NSX UI