Error: REPO_SYNC FAILED status on newly redeployed NSX Manager nodes

Products

• VMware NSX

Issue/Introduction

1. On VMware NSX Manager appliances, the REPO_SYNC service may enter a FAILED state specifically after one or more Manager nodes have been redeployed or replaced within an existing cluster.
2. While the Cluster status may show as healthy, the repository synchronization remains unsuccessful, and clicking the Resolve button in the UI does not clear the error.
3. Reviewing the logs, it is observed that the install-upgrade service is enabled on one of the newly deployed Manager nodes which doesn't have the required directory structure in the /repository folder.

Environment

VMware NSX

Cause

This issue occurs when newly redeployed Manager nodes lack the specific version-controlled directories or files in the /repository path that are present on the original orchestrator node. Because the install-upgrade service relies on local file availability for synchronization, the new nodes cannot automatically pull the full repository content if the directory structure is missing or permissions are incorrect.

Resolution

As the current orchestrator is having issues with successfully syncing with the repository itself, it may be beneficial to change which NSX Manager repository is identified as the "main" repository. If another Manager node exists in the Cluster, which has the correct directory structure, it can be set to be the orchestrator node.

1. 1. Open a putty session to each NSX Manager and log in with the admin credentials
   2. Run the following command on each NSX Manager stop service install-upgrade
      • Confirm that each NSX Manager has the install-upgrade service showing a status of stopped by running the command get service install-upgrade. The output for each Manager should be similar to this:
        
   3. Once each NSX Manager shows the service has stopped select an NSX Manager to be the new "main" repository (preferably a Manager that has not been replaced/redeployed)
      • In the putty session for the selected Manager run the command set repository-ip and when prompted type yes to continue. The output will look similar to:
        
      • Note that even though in this specific situation the REPO_SYNC status is currently failed, the above process can be done as the current repository is not synching correctly
   4. Wait a few minutes after this process to then bring the install-upgrade state back up
      • To do this, from the putty sessions on each NSX Manager run the command start service install-upgrade and then verify that service is now running by running the command get service install-upgrade. Confirm that each NSX Manager shows a service state of running before continuing
   5. Attempt another RESOLVE for the REPO_SYNC for one of the NSX Managers (NSX UI > System > Appliances > View Details on one of the NSX Managers and click RESOLVE for the Failed REPO_SYNC) 
      • If the sync is successful on the NSX Manager, attempt the RESOLVE on the other Managers

Additional Information

Additional KB articles that might help with troubleshooting if above steps fail to resolve the issue:

After replacing Managers or while running Upgrade prechecks, Repo_Sync is Failed

NSX REPO_SYNC failure on all 3 manager nodes