"Please ensure that Repository is in sync with the other nodes" Error seen when attempting to upload PUB (Pre-Check Upgrade Bundle) file for an NSX upgrade

Products

• VMware NSX

Issue/Introduction

• During an NSX upgrade, when uploading the PUB file, the following error is displayed and you are unable to proceed:

Unable to start upgrade-coordinator upgrade. The version null is not recognized by the manager. Please ensure that the Repository is in sync with the other nodes.

• Prior to this error, there was an issue with repository and the manager where rebooted.
• An alarm is present for NSX platform API certificates for the manager have expired.
• On the manager which is doing the upgrade, the Orchestrator node, the following error is seen in the /var/log/proton/nsxapi.log:

INFO RepoSyncStartUpThread-1762417059443 RepoSyncFileHelper 4514 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Command to check if remote file exists for https://<Manager-IP>:443/repository/4.2.3.1.0.24954727/UC/localized_eula/EULAContentMessages_ja.properties returned result CommandResultImpl [commandName=null, pid=19226, status=FAILED, errorCode=60, errorMessage=curl_wrapper: (60) certificate has expired

• Checking the contents of the /repository directory of all 3 managers as root user, this manager is missing the New version directory:

/repository:
total 36K
drwxr-xr-x 26 root root     4.0K Nov  6 08:19 ..
drwxrwx---  5 uuc  grepodir 4.0K Oct 28 11:54 .
-rwxrw----  1 uuc  grepodir   18 Oct  3  2023 current_version
drwxrwx---  7 uuc  grepodir 4.0K Oct  3  2023 4.1.1.0.0.22224312 -> Old to version
drwxrwx---  7 uuc  grepodir 4.0K Apr 25  2023 4.0.1.1.0.20598726 -> Current version
drwxrwx---  2 uuc  grepodir  16K Jun 27  2022 lost+found

• Contents of the other managers shows the to version:

/repository:
total 36K
drwxr-xr-x 26 root root     4.0K Nov  6 08:23 ..
drwxrwx---  5 uuc  grepodir 4.0K Nov  6 08:19 .
drwxrwx---  6 uuc  grepodir 4.0K Sep 11 09:22 4.2.3.1.0.24954727 -> New to version
-rwxrw----  1 uuc  grepodir   18 Oct  3  2023 current_version
drwxrwx---  7 uuc  grepodir 4.0K Oct  3  2023 4.1.1.0.0.22224312 -> Current version
drwxrwx---  2 uuc  grepodir  16K Jun 27  2022 lost+found

Note: If repository sync is working correctly, each manager should have the same view.

Environment

VMware NSX

Cause

The API certificates had expired, this lead to the repository sync issue, but prior to the certificates being replaced, the NSX managers where rebooted, this lead to the orchestrator service moving to the other NSX manager, this NSX manager had an issue with repository sync and as such did not have all the file.

Resolution

Workaround:

On one of the managers that has the complete files in the /repository directory, preferable the original orchestrator node, if known, as admin user run the command:

set respoitory-ip

This will move the UC (Upgrade coordinator) service to this manager, the one the command is run on.

Then in the UI, proceed with the PUB file upload.