Alarm GM to LM sync warning for non existent site/location after failed onboarding

Products

• VMware NSX

Issue/Introduction

• You receive a Alarm for GM to LM sync warning with reference to a site that is not onboarded or failed to onboard. 
• In the Alarm description you will see reference to the failed remote site:
  Data synchronization between sites <Global NSX Manager>(########-####-####-####-########f034) and <failed-remote-site>(########-####-####-####-########cff1) failed for the FlowIdentifier{role='Policy', nameSpace='LM_2_GM_NOTIFICATION'}.
• On NSX Manager, you may see the following entries in the following log file /var/log/async-replicator/ar.log
  2024-05-29T10:51:02.800Z  WARN MetricsRefresh-0 Replicator 3781 - [nsx@6876 comp="global-manager" level="WARNING" subcomp="async-replicator"] AR leader or its IP address is not known for remote-site ########-####-####-####-########cff1

Environment

VMware NSX 

Cause

A failed onboarding for a Local Manager may leave some configurations or references on the Global Manager. 

Resolution

If onboarding of the local site to the global manager fails, the following API call should be run to clean stale site entries in the Global Manager 

curl -u "<username>:<password>" -X POST -ik https://<primary-gm-ip>/api/v1/sites?action=offboard_remote -H "Content-Type: application/json" -d '{"credential": {"ip": "<IP>", "port":443, "username": "<USERNAME>", "password": "<PASSWORD>", "thumbprint": "<THUMBPRINT>"}, "site_id": "########-####-####-####-########cff1"}'

"<username>:<password>" from the Global Manager

{"ip": "<IP>", "port":443, "username": "<USERNAME>", "password": "<PASSWORD>", "thumbprint": "<THUMBPRINT>"} from the Local Manager

This resolution is for stale site entries in Global Manager, not for fully onboarded sites. If needing to offboard a non-stale LM site, please refer to Remove a Location.

Additional Information

Restarting the async-replicator service can also help in fixing these alarms:
restart service async_replicator  

GM to LM data synchronization warning alarm
https://knowledge.broadcom.com/external/article?articleNumber=345832

NSX Local Manager configuration import fails in a Federation environment
https://knowledge.broadcom.com/external/article?articleNumber=318522