Upgrading a Brownfield-imported domain fails during the upgrade pre-check phase due to an unsupported upgrade path for NSX.

Products

VMware SDDC Manager / VCF Installer VMware NSX VMware Cloud Foundation

Issue/Introduction

During a Brownfield-import of vCenter, the latest compatible version of NSX will be deployed. When trying to upgrade this domain to a version of NSX which is not in a supported upgrade path as per VMware Interoperability Matrix, upgrade precheck will fail.

Example:
During Brownfield Import of vCenter on VCF 5.2.2, by default, NSX Manager with version 4.2.3.3(Latest patch release at the time of publishing the KB) gets deployed.
When customer wants to upgrade VCF 5.2.2 to VCF 9.0.1, NSX upgrade pre-checks fail complaining that NSX can't be upgraded to 9.0.1.

Environment

VMware Cloud Foundation 5.2.x and above

Resolution

Following are the options to resolve this issue:

Wait until next release of VCF (at the time of writing this document, next VCF release will be 9.1.0) that supports the upgrade path with the specific NSX version.
Un-prepare the vCenters clusters for NSX and remove the domain from inventory and then deploy custom NSX while importing the vCenter that supports the upgrade path to VCF 9.0.1.

This document will focus on the instructions for the Option#2 mentioned above.

Steps

Note down the domain details. Go to SDDC Manager UI → Inventory → Workload Domains → Select the Domain. From the URL link, note down the domain ID.
. Go to Services tab and note down vCenter FQDN and NSX Manager FQDN.
Un-prepare vCenter clusters for NSX
1. In NSX Manager UI, Go to System → Fabric → Hosts
2. Under Clusters tab, select the clusters in the specified vCenter
3. Select Remove NSX option. It will prompt for confirmation
4. Once confirmed, uninstallation will start. Wait for the uninstallation to get completed. All hosts are will have `Not Configured` NSX Configuration once uninstallation completes.
Go to vSphere Client UI -> Cluster -> Updates -> Hosts -> Image -> Check Compliance to ensure that the cluster is compliant. If Cluster is not compliant, remediate the cluster.
Remove Compute Manager:
1. In NSX Manager UI, Go to System → Fabric → Compute Managers
2. Click on 3 dots on left side of the compute manager for the particular vCenter.
3. Click on Delete option
Delete NSX VM(s) in the vCenter
1. Go to vCenter UI, Select the NSX Manager VM → Actions → Power → Power Off → Confirm power Off
2. Once powered off, select the NSX Manager VM → Actions → Delete from Disk → Confirm Delete
Remove the domain from the SDDC-M inventory
1. Remove Nsxt Cluster for the domain
  Note: Run this only if the NSX Manager is used by only this domain. Else Proceed with the next step.
  Run following API on SDDC Manager:
  - curl -i -X DELETE http://localhost/inventory/extensions/vi/nsxtclusterdomains/{domain-id}
2. Remove hosts for the domain
  Run following APIs on SDDC Manager:
  - Get token : TOKEN=$(curl -H 'Content-Type:application/json' https://localhost/v1/tokens -d '{"username" : "admin@local","password":"<Password>"}' -k | jq -r '.accessToken')
  - Get hosts : curl -k -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -X GET https://localhost/v1/hosts?domainId={domain-id}
  - Note down the id for all hosts in the domain.
  - Create a new file : ESXiIds.json in SDDC Manager
  - Add the ESXi Ids obtained for the domain in ESXiIds.json in the following format:
```
            [
                  {
                        "id": "host-id-1"
                  },
                  {
                        "id": "host-id-2"
                  },
                  {
                        "id": "host-id-3"
                  },
                  {
                        "id": "host-id-4" 
                  }
            ]
```
  - Remove hosts : curl -s -X DELETE http://localhost/inventory/extensions/decommission/hosts -H "Content-Type: application/json" -d @ESXiIds.json
Run following API on SDDC Manager to delete domain:
curl -s -X DELETE http://localhost/inventory/extensions/vi/domains/{domain-id}
Run following API on SDDC Manager to "Get Domains" to confirm that the domain is deleted
curl -i -X GET http://localhost/inventory/extensions/vi/domainInventory
Remove the SDDC Manager plugin registration in the vCenter
1. Go to https://<vCenter FQDN>/mob and login with vCenter credentials
2. Go to content → ExtensionManager
3. You will be able to see extensionList["com.vmware.sddcManager"] in the VALUE column for extensionList in Properties Section
4. Go to Methods Section and select UnregisterExtension
5. Give com.vmware.sddcManager as Parameters value for extensionKey
6. Trigger Invoke Method on Right Bottom Corner of the pop-up.
7. Close the pop-up. Refresh the ExtensionManager window and make sure that above extensionKey is removed.
Update SDDC Manager Application Configs for deploying custom NSX during Import/Convert, Refer KB : https://knowledge.broadcom.com/external/article?articleNumber=429205
Retry Brownfield Import workflow for the vCenter.
Once import is Successful, retry the upgrade.