NSX Manager Pre-check error <repository sync has failed>

Products

• VMware NSX

Issue/Introduction

• NSX Manager upgrade failed in pre-check.
  Repository sync is not complete on <NSX-Node-1>, <NSX-Node-2>, <NSX-Node-3>. Please wait for repository sync to complete, or resolve the errors if repository sync has failed.


  
  
• In the NSX Manager UI , navigate to System -> Appliances and then "View Details" for each manager appliance, REPO_SYNC is in Failed state.
• The REPO_SYNC > RESOLVE workflow doesn't help and fails.
  
  
• Below logs can be seen in nsxapi.log showing checksum mismatch : [/var/log/proton/nsxapi.log]
  
  

  <timestamps>  INFO task-executor-0 DownloadTaskMonitor 4467 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] File Download failed with result CommandResultImpl [commandName=manifest, pid=2422643, status=FAILED, errorCode=0, errorMessage=/repository/4.1.0.2.0.21761691/metadata/manifest : Downloaded file checksum 420d0b687557803390186929cdcb14bcec0725c9164febace61b28080d520daf does not matches with expected checksum 3b86336f63096d54c61d38fdab0de93980fca281b043b16a47c8fd9a04d4a59c., commandOutput=null]

Environment

VMware NSX >> Source Version >> 4.1.0.2.0.21761691

Cause

A MD5 checksum validation failure occurred for the manifest file located in the /repository/4.1.0.2.0.21761691/metadata/ directory. The calculated checksum of the local file did not match the expected value required by the NSX Manager, preventing the system from verifying the integrity of the repository metadata.

Resolution

To fix the issue, follow the below steps :

• Download nsx-unified-appliance-<version>.ova file following these instructions: Download Broadcom products and software. The downloaded version must match the repository version specified in the 'Issue' section above.
• Deploy this NSX Manager as a separate appliance in vCenter and do not connect to the NSX Cluster.
• Keep a backup of files under /repository/<version>.

• From this newly deployed NSX Manager, copy the /repository/<version> directory to all 3 existing managers missing the directory.
  
  

  Open an SSH session to the newly deployed NSX Manager.
  scp -r /repository/<version> root@<IP of NSX Manager [other 3 nodes]>:/repository/

• As root user, run the command /opt/vmware/proton-tomcat/bin/reposync_helper.sh on all the 3 existing managers, not the newly deployed one.
  
  
• From the NSX UI Resolve the REPO_SYNC on the orchestrator node: System -> Appliances -> View Details and click Resolve for REPO_SYNC and wait for this to complete.
  

  To identify the Orchestrator node, log into any Manager as admin and run: 

  nsx-mngr> get service install-upgrade
Service name:      install-upgrade
Service state:     stopped
Enabled on:        <NSX-IP>   <<< orchestrator node

• Now resolve the repo-sync failure on the other 2 nodes, from System-> Appliances page and wait for this to complete.
• Proceed to upgrade NSX Manager as the pre-check will be successful now.
• The newly deployed NSX Manager can now be powered off and deleted once the REPO_SYNC is working.

Additional Information

After replacing Managers or while running Upgrade prechecks, Repo_Sync is Failed