"Unable to start upgrade-coordinator upgrade" Error displayed when importing a PUB file in NSX for an upgrade

Products

• VMware NSX

Issue/Introduction

• During the NSX upgrade, the PUB (Pre-Check Upgrade Bundle) file is being uploaded and displays the error:

Unable to start upgrade-coordinator upgrade. The version null is not recognized by the Manager. Please ensure that Repository is in sync with the other nodes.

• Checking each manager repository partition as root user, we can see the files are not in sync across all 3 managers:

Manager1: /repository:
total 36K
drwxr-xr-x 26 root root     4.0K Nov  6 08:19 ..
drwxrwx---  5 uuc  grepodir 4.0K Oct 28 11:54 .
-rwxrw----  1 uuc  grepodir   18 Oct  3  2023 current_version
drwxrwx---  7 uuc  grepodir 4.0K Oct  3  2023 4.1.1.0.0.22224312
drwxrwx---  7 uuc  grepodir 4.0K Apr 25  2023 4.0.1.1.0.20598726
drwxrwx---  2 uuc  grepodir  16K Jun 27  2022 lost+found

Manager2: /repository:
total 36K
drwxr-xr-x 26 root root     4.0K Nov  6 08:23 ..
drwxrwx---  5 uuc  grepodir 4.0K Nov  6 08:19 .
drwxrwx---  6 uuc  grepodir 4.0K Sep 11 09:22 4.2.3.1.0.24954727
-rwxrw----  1 uuc  grepodir   18 Oct  3  2023 current_version
drwxrwx---  7 uuc  grepodir 4.0K Oct  3  2023 4.1.1.0.0.22224312
drwxrwx---  2 uuc  grepodir  16K Jun 27  2022 lost+found

Manager3: /repository:
total 36K
drwxr-xr-x 26 root root     4.0K Nov  6 08:50 ..
drwxrwx---  5 uuc  grepodir 4.0K Nov  6 08:29 .
drwxrwx---  6 uuc  grepodir 4.0K Sep 11 09:22 4.2.3.1.0.24954727
-rwxrw----  1 uuc  grepodir   18 Oct  3  2023 current_version
drwxrwx---  7 uuc  grepodir 4.0K Oct  3  2023 4.1.1.0.0.22224312
drwxrwx---  2 uuc  grepodir  16K Jun 27  2022 lost+found

Note: Manager1 above is not consistent with the other 2 nodes, it still has the old versions.

• In the past there was an error with the repository and the managers where rebooted to resolve it.
• In the NSX manager log /var/log/proton/nsxapi.log we can see certificate errors:

INFO RepoSyncStartUpThread-1762417059443 RepoSyncFileHelper 4514 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Command to check if remote file exists for https://<manager-IP>:443/repository/4.2.3.1.0.24954727/UC/localized_eula/EULAContentMessages_ja.properties returned result CommandResultImpl [commandName=null, pid=19226, status=FAILED, errorCode=60, errorMessage=curl_wrapper: (60) certificate has expired
ERROR RepoSyncStartUpThread-1762417059443 RepoSyncServiceImpl 4514 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP21057" level="ERROR" subcomp="manager"] Unable to calculate current node repo sync status.

• Checking the UI, System, Certificates, there are expired certificates for the NSX managers API and/or the mp-cluster.

Environment

VMware NSX

Cause

The Upgrade coordinator service was unable to upgrade, due to the repository not being in sync between the managers.
The repository sync failed due to the expired manager certificates.

Resolution

Please see details in the KB REPO_SYNC is in Fail state on one or more NSX Managers and NSX API certificates are expired on how to replace the certificates and get the repository sync working again.

Once the certificates are replaced and the repository sync is working again, re-upload the NSX PUB file to proceed with the upgrade.