NSX REPO_SYNC failure on all 3 manager nodes

Products

VMware NSX

Issue/Introduction

In the NSX UI, there is a REPO_SYNC failure for each of the 3 manager nodes.
The nsxapi.log on the nodes report that the manager is reaching out to multiple repositories for sync:

#########DATE####### INFO RepoSyncThread-############# RepoSyncServiceImpl #### SYSTEM [nsx@#### comp="nsx-manager" level="INFO" subcomp="manager"] Version extracted from NSX_MANAGER_REPO_LIST_PUB_4.2.1.4.0.24771817 = 4.2.1.4.0.24771817
#########DATE####### INFO RepoSyncThread-############# RepoSyncServiceImpl #### SYSTEM [nsx@#### comp="nsx-manager" level="INFO" subcomp="manager"] Version extracted from NSX_MANAGER_REPO_LIST_4.2.1.3.0.24533884 = 4.2.1.3.0.24533884

NOTE: The above snippet is generic and the versions mentioned here are for reference only.

The output of "GET https://<NSX_MGR3>/api/v1/fabric/modules" report that there are multiple repositories under "ManagerNodeVMFabricModule":

"key" : "NSX_MANAGER_REPO_LIST_PUB_4.2.1.4.0.24771817",
"value" : "/repository/4.2.1.4.0.24771817/metadata/prechecks-****"
"key" : "NSX_MANAGER_REPO_LIST_4.1.0.2.0.21761691",
"value" : "/repository/4.1.0.2.0.21761691/metadata/****"
"key" : "NSX_MANAGER_REPO_LIST_4.2.1.3.0.24533884",
"value" : "/repository/4.2.1.3.0.24533884/metadata/prechecks-****"

Cause

This is a known issue impacting VMware NSX caused by missing files within the /repository directory within each NSX Manager due to a previous upgrade leaving behind stale entries and the manager nodes looking for the stale directories to sync.

Resolution

NOTE: Ensure to make a full backup of the managers before you proceed further, and confirming the passphrase for the backup is known.

Run "stop service install-upgrade" on all 3 managers as admin user.
Get the output of "GET https://<NSX_MGR#>/api/v1/fabric/modules" and in the output search for the string "ManagerNodeVMFabricModule".
Verify that its corresponding "custom_data" field contains references to the version you are clearing out from repository, and pick the ID of the module listing this version as a requirement from the end of the same record, which should show in this form:
"resource_type": "FabricModule",
"id": "<module_UUID>",
"display_name": "<module_UUID>",
"_create_time": #############,
"_create_user": "system",
"_last_modified_time": #############,
"_last_modified_user": "UC",
"_system_owned": false,
"_protection": "NOT_PROTECTED",
"_revision": 2
}
Run "GET https://<NSX_MGR3>/api/v1/fabric/modules/<module_UUID>" and copy the output of the query into a text editor, and delete the keys matching with the name NSX_MANAGER_REPO_LIST_<version> or NSX_MANAGER_REPO_LIST_PUB_<version> from the custom_data list of entries.
- NOTE: Ensure to only remove the erroneous version entries and to leave the entry matching with the current version of NSX.
Input the edited contents as raw payload into a query: PUT "https://<NSX_MGR3>/api/v1/fabric/modules/<module_UUID>" along with the edited body of the previous command without the stale entries.
Repeat the query GET https://<NSX_MGR3>/api/v1/fabric/modules/<module_UUID> API call in order to confirm the contents do not reference the erroneous NSX version anymore.
From the NSX UI, Resolve the REPO_SYNC on the orchestrator node: System -> Appliances -> View Details click "Resolve" for REPO_SYNC.
Once completed, repeat steps 2-7 for each of the other 2 Managers and this should bring the REPO_SYNC status back to Success state.

Additional Information

If you are contacting Broadcom support about this issue, please reference this KB and provide the following:

The current version of NSX .
The version being upgraded to.
The state of the REPO_SYNC on all three managers

Handling Log Bundles for offline review with Broadcom support