NSX Manager REPO_SYNC Fails Due to Upgrade Coordinator Version Mismatch

Products

• VMware NSX

Issue/Introduction

• This issue occurs following a previously initiated and subsequently cancelled NSX upgrade attempt, where all associated target version binaries were successfully purged from the system repository of the primary node.
  
  
• All or secondary nodes in the VMware NSX 4.x management cluster fail repository synchronization (REPO_SYNC), leading to nodes reporting as "not configured" in the CLI (get nodes) or displaying degraded (yellow/red) health states in the user interface.
  

  nsx-manager> get nodes
  UUID                                  Type IP Address      IPv6 Address        Display Name                                  
  887b####-####-####-####-########5650  mgr  ##.##.##.1     not configured       nsx-manager-1
  95e8####-####-####-####-########aefb  mgr  ##.##.##.2     not configured       nsx-manager-2
  0a03####-####-####-####-########5783  mgr  ##.##.##.3     not configured       nsx-manager-3

   

• /var/log/syslog or /var/log/cm-inventory/cm-inventory.log file records failures stating: "Failed to sync repository from primary manager node" deleted version directory. 
  

  <Time_Stamp> <mngr_IP_address> cm-inventory [nsx@6876 comp="nsx-manager" level="ERROR" subcomp="repo-sync"] Failed to sync repository from primary manager node: HTTP 404 Not Found for file /repository/<DELETED_VERSION>/Manager/ovf/nsx-unified-appliance-<DELETED_VERSION>.###

 

• Executing the upgrade summary check reveals a version divergence between the active system version and the Upgrade Coordinator engine (deleted version):

  nsx-manager# curl -k -u admin -X GET "https://localhost/api/v1/upgrade/summary"
  {
    "system_version" : "<4.#.#.#.#.#########>",
    "upgrade_coordinator_version" : "<4.X.X.X.X.XXXXXXXXX>",
    "pre_upgrade_bundle_version" : "<4.X.X.X.X.XXXXXXXXX>",
    "upgrade_status" : "NOT_STARTED",
    "upgrade_coordinator_updated" : false
  }

Environment

VMware NSX

Cause

An NSX upgrade bundle was manually removed from disk while stale ManagerNodeVMFabricModule metadata entries and cached upgrade-coordinator service version registers remained pointing to the deleted build.

This issue occurs when an NSX upgrade bundle directory is manually deleted from the disk without removing the corresponding ManagerNodeVMFabricModule metadata entries from the NSX database. The upgrade-coordinator service retains stale pointers to the deleted build, causing secondary nodes to fail during repository convergence.

Resolution

To resolve this issue, manually sanitize the fabric module metadata and force-align the Upgrade Coordinator version:

• Log in to the primary NSX Manager node as root via SSH.
  
  
• Fetch the active fabric module metadata JSON to identify the module ID: curl -k -u admin -X GET "https://localhost/api/v1/fabric/modules" | grep -A 100 "ManagerNodeVMFabricModule" | grep -m 1 '"id"'
  
  
• Export the module data to a file for editing (replace <ModuleID> with the ID found in step above): curl -k -u admin -X GET "https://localhost/api/v1/fabric/modules/<ModuleID>" > /tmp/fabric_module.json
  
  
• Create a backup of the file: cp /tmp/fabric_module.json /tmp/fabric_module.json.bak
  
  
• Edit /tmp/fabric_module.json to remove all key-value blocks and deployment specifications referencing the deleted upgrade version. Ensure the resulting file maintains valid JSON syntax.
  
  
• Push the updated JSON payload back to the database : curl -k -u admin -H "Content-Type: application/json" -X PUT -d @/tmp/fabric_module.json "https://localhost/api/v1/fabric/modules/<ModuleID>"
  
  
• Retrieve the primary node UUID and force-align the Upgrade Coordinator (UC) version to the currently running system version (replace <NodeUUID> and <ActiveVersion> with your environment's values): curl -k -u admin -H "Content-Type: application/json" -X POST -d '{"product_version": "<ActiveVersion>"}' "https://localhost/api/v1/cluster/<NodeUUID>/node/upgrade/uc"
  
  
• Wait 2 to 5 minutes for the upgrade-coordinator service to restart.
  
  
• Verify that system_version and upgrade_coordinator_version are now aligned : curl -k -u admin -X GET "https://localhost/api/v1/upgrade/summary
  
  
• In the NSX Manager UI, navigate to System > Appliances click RESOLVE on the primary manager appliance first, then click RESOLVE on all secondary nodes to complete synchronization.

Additional Information

UC is not updated on node" NSX Manager upgrade pre-check error.

Check for the version sync status of UC on all the MP nodes.