NSX Upgrade Precheck Fails with Repository Error
search cancel

NSX Upgrade Precheck Fails with Repository Error

book

Article ID: 402081

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

When attempting to run a precheck prior to upgrading the NSX environment the following error is seen:

Unable to connect to File /repository/<NSX_Version_Here>/UC/bundle_type_pub on source <NSX_Manager_Here>. Please verify that file exists on source and install-upgrade service is up.

After the error is received, when reviewing the NSX Managers within the NSX UI the state for REPO_SYNC status (System > Appliances > View Details on the NSX Manager) is showing as FAILED:

Please note errors and screenshots here may differ slightly based on environment configurations.

Environment

VMware NSX 4.1.2.0

Cause

This is a known issue impacting VMware NSX. It can be due to a few things:

  • The Install-Upgrade service is not running.
  • Missing files within the /repository directory within each NSX Manager.
  • The NSX Managers Upgrade Coordinator is showing different versions (for example one Manager may show that its Upgrade Coordinator 4.1.2.1.0 while another may show its Upgrade Coordinator is 4.2.2.1.0).
  • The Orchestrator NSX Manager repository may not be properly syncing the necessary files from the .PUB with the other Managers.

Resolution

This issue is resolved in VMware NSX 4.2.0 or later.

Depending on what issue is the root cause will determine which path to resolution should be applied. 

Install-Upgrade Service Is Not Running

Verify if the service is up and running by performing the steps below:

  1. Open an SSH session to each NSX Manager and log in with the admin credentials
  2. Run the command get service install-upgrade to confirm whether this service is running or not
    • The output should look similar to the one below:
    • The IP listed next to "Enabled on" will identify which Manager is the orchestrator
  3. If the state of the service shows as stopped, attempt to start the service by running the command restart service install-upgrade and wait a few minutes before running the get service install-upgrade command to verify if the service is running or not.
    • If the service started successfully and continues to run, go back to the NSX UI > System > Appliances > View Details on one of the NSX Managers and click RESOLVE for the Failed REPO_SYNC to see if the Failed status changes to success. If this REPO_SYNC still reports a Failed status, please move onto the Possible Missing Files section below.
    • If the service does not start or seems to intermittently stop/start please refer to NSX install-upgrade service goes down intermittently or open a Support Ticket with VMware by Broadcom Support. 

Possible Missing Files

If the error is advising that a specific file is missing or cannot be connected to, first verify if the file is actually missing.

Below are few examples of possible error messages: 

  • Unable to connect to File /repository/<NSX_version>/UC/bundle_type_pub on source <nsx_manager_name_here>. Please verify that file exists on source and install-upgrade service is up.
  • Unable to connect to File /repository/<NSX_version>/UC/HostComponent/rhe_179_x86_64_baremetal_server/upgrade.sh on source <nsx_manager_name_here>. Please verify that file exists on source and install-upgrade service is up.

Within the syslog of the NSX Manager (/var/log/syslog) the following may be present:

  • 2025-06-11T22:29:12.070Z  INFO http-nio-###.#.#.#-####-exec-2 UpgradeQueryServiceImpl 3610031 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Uc Upgrade status is FAILED hence creating bundle status from UcUpgradeStatus{percentage=0, status=FAILED, progressMessages=[], errorMessages=[[##.###.#.###] Unable to connect to File /repository/4.2.2.1.0.24765084/Manager/dry-run/corfu-server-metadata-editor.jar on source <nsx_manager_name_here>. Please verify that file exists on source and install-upgrade service is up., [##.##.##.###] Unable to connect to File /repository/<NSX-version>/Manager/dry-run/corfu-server-metadata-editor.jar on source <nsx_manager_name_here>. Please verify that file exists on source and install-upgrade service is up.], step=REPO_SYNC, upgradeBundleType='PUB'}
  • [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] response: Response:: status code:200 OK status: body:123
      "status" : "FAILED",
      "status_message" : "",
      "failure_message" : "Unable to connect to File /repository/<NSX-version>/Manager/dry-run/corfu-server-metadata-editor.jar on source <nsx_manager_name_here>. Please verify that file exists on source and install-upgrade service is up.",
      "failure_code" : 21057

To verify if the file is missing or not perform the following:

  1. Open a putty session to the NSX Manager specified in the error and log in with root credentials.
  2. Run the command ls -l /repository and hit enter to review the directories present.
    • The output should look similar to this (please note the version numbers below are just examples and may differ by environment).
    • There should be a to file (this is the version that the environment is being upgraded to and is represented by the blue box above).
    • There should be a from file (this is the version that the environment is being upgraded from and is represented by the green box above).
    • If the output only shows one of the version files or there are no files listed please refer to After replacing Managers or while running Upgrade prechecks, Repo_Sync is Failed.
  3. Run the command cd <version_listed_in_error>/... to move into the directory specified from the error (for example: this command would be cd <version_listed_in_error>/Manager/dry-run/ as the error stated the path to the file was /repository/<version_listed_in_error>/Manager/dry-run).
    We manually copied the missing directories/files to the impacted NSX managers, by following below commands.

    Copy /repository/<version_listed_in_error>/Manager/dry-run/ directory to the impacted NSX managers.

    Open an SSH session to the manager where all the required files are present, and copy the missing files to the impacted NSX managers.
    #scp -r /repository/<version_listed_in_error>/Manager/dry-run/ root@<NSX-Manager-IP>:/repository/

    This command copies the /repository directory recursively to the root directory (/) of host A.B.C.D.
    Now the user,  group, and permission will need to be check and corrected.

    This will recursively set the user and group:
    #chown -R uuc:grepodir /repository

    This will recursively set the required permissions:
    #chmod -R 770 /repository

  4. After moving into the directory run the command ls -l to review the items within that path and verify if the file from the error is present.
    • For example if the error stated the following path /repository/<NSX-version>/Manager/dry-run/corfu-server-metadata-editor.jar then confirmation that the file corfu-server-metadata-editor.jar is needed.
    • If the file is missing please open a Support Ticket with VMware by Broadcom Support.
  5. If the file within the error is present within the directory proceed to the next portion Upgrade Coordinator Versions Differ Between NSX Managers below.

Upgrade Coordinator Versions Differ Between NSX Managers

Utilizing an API platform such as Postman run the following GET commands for each NSX Manager.

  1. GET https://<NSX Manager IP Here>/api/upgrade/summary
  2. Compare the upgrade_coordinator_version section and target_version between the managers. 
    • If the versions differ between NSX Managers, open a support ticket with VMware by Broadcom Support.
  3. Go back to the NSX UI > System > Appliances > View Details on one of the NSX Managers and click RESOLVE for the Failed REPO_SYNC
    • If the status changes to success, perform the same process of selecting RESOLVE on the other NSX Managers (one at a time) and wait until all Managers report a Success for REPO_SYNC:
    • If the status still reports as Failed, proceed to the next section: NSX Orchestrator Unable To Sync With Itself

NSX Orchestrator Unable To Sync With Itself

As the current orchestrator is having issues with successfully syncing with the repository itself, it may be beneficial to change which NSX Manager repository is identified as the "main" repository.

    1. Open a putty session to each NSX Manager and log in with the admin credentials
    2. Run the following command on each NSX Manager stop service install-upgrade
    3. Confirm that each NSX Manager has the install-upgrade service showing a status of stopped by running the command get service install-upgrade. The output for each Manager should be similar to this:
    4. Once each NSX Manager shows the service has stopped select an NSX Manager to be the new "main" repository (preferably a Manager that has not been replaced/redeployed)
    5. In the putty session for the selected Manager run the command set repository-ip and when prompted type yes to continue. The output will look similar to:
      • Please note that even though in this specific situation the REPO_SYNC status is currently failed, the above process can be done as the current repository is not synching correctly
    6. Wait a few minutes after this process to then bring the install-upgrade state back up
      • To do this, from the putty sessions on each NSX Manager run the command start service install-upgrade and then verify that service is now running by running the command get service install-upgrade. Confirm that each NSX Manager shows a service state of running before continuing
    7. Attempt another RESOLVE for the REPO_SYNC for one of the NSX Managers (NSX UI > System > Appliances > View Details on one of the NSX Managers and click RESOLVE for the Failed REPO_SYNC) 
      • If the sync is successful on the NSX Manager, attempt the RESOLVE on the other Managers
      • If the sync fails again, continue to step 8
    8. Attempt to re-upload the precheck file from the NSX UI: System > Upgrade > Check Upgrade Readiness > Select Local File > ensure the file downloaded ends with .pub and once selected click Upload
      • If the precheck upload is successful this time and no errors are present continue with running the prechecks
      • If the precheck upload is still failing with the Unable to connect File....  message, please open a Support Ticket with VMware by Broadcom for further assistance

Note: 

If none of the above steps fix the issue try workaround options based on deployment of Managers in After replacing Managers or while running Upgrade prechecks, Repo_Sync is Failed. (Workaround may need to be applied to all managers not just the Orchestrator node in the Option: Deploy MUB file in /repository: workaround.)

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

  1. The current version of NSX
  2. The version being upgraded to
  3. Current state of the REPO_SYNC on all NSX Managers
  4. What steps have been done so far with regard to troubleshooting this issue
  5. Log Bundles for:
    • All NSX Managers and ensure that the Log age (days) option is set to All.  
Powered by Wolken Software