Unable to upgrade imported workload domain in VCF 9.1.1 when SDDC Manager reports a partially upgraded NSX
search cancel

Unable to upgrade imported workload domain in VCF 9.1.1 when SDDC Manager reports a partially upgraded NSX

book

Article ID: 457950

calendar_today

Updated On:

Products

VMware NSX VMware SDDC Manager / VCF Installer

Issue/Introduction

  • On Plan Domain Upgrade page in VCF Operations the following is seen:

    You cannot select this target version for the following reasons. Select a different target version to proceed.
    Domain <UUID> with nsx cluster <NSX VIP FQDN> is partially upgraded.
    Unable to verify the compatibility for the following product versions. lease check the product documentation before proceeding to upgrade: NSX_T_MANAGER <version>. You may use http://interopmatrix.broadcom.com/interoperability.

  • On the same page, the Next button is greyed out.
  • Running the GET /api/v1/upgrade-mgmt/plan API call against the NSX manager returns results similar to the following (truncated):

    {
      "upgrade_status": "PAUSED",
      "source_version": "<version>",
      "target_version": null,

  • The Upgrade page in the NSX UI may show that no upgrade is in progress. If the install-upgrade service is restarted on all NSX Manager nodes, this may change to show an older upgrade that successfully completed as still being in progress.
  • If the NSX UI shows an upgrade as being in progress, NullPointerException error is generated on any attempt to resume it. Messages similar to the following will be seen in the /var/log/upgrade-coordinator/upgrade-coordinator.log file:

    2026-09-18T20:38:21.578Z  INFO http-nio-127.0.0.1-7442-exec-1 UpgradeHistoryPersistenceServiceImpl 1102148 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] No upgrade history found with from version <version> , target version null and upgrade status STARTED
    2026-09-18T20:38:21.578Z  INFO http-nio-127.0.0.1-7442-exec-1 UpgradeHistoryPersistenceServiceImpl 1102148 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [UPS] Setting Upgrade history with fromVersion <version> toVersion null startTime 1789763901574 upgradeStatus STARTED
    2026-09-18T20:38:21.578Z  INFO http-nio-127.0.0.1-7442-exec-1 UpgradeServiceImpl 1102148 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Setting current upgrade status as PAUSED
    2026-09-18T20:38:21.578Z  INFO http-nio-127.0.0.1-7442-exec-1 UpgradeServiceImpl 1102148 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] In setOverallUpgradeStatus for PAUSED, upgradeExecutionContext.getUpgradeStatus(): PAUSED
    2026-09-18T20:38:21.589Z ERROR http-nio-127.0.0.1-7442-exec-1 UpgradeServiceImpl 1102148 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30078" level="ERROR" subcomp="upgrade-coordinator"] Error while continuing the upgrade for component MP
    java.lang.NullPointerException: null

  • Message similar to the following are present in the /var/log/lcm/lcm-debug.log file on the SDDC Manager VM:

    2026-09-24T14:11:05.400Z DEBUG vcf_lcm 561923 [sddcm@4413 traceId="########" spanId="c01a" threadId="http-nio-127.0.0.1-7400-exec-6" class="c.v.e.s.l.s.v.i.PartiallyUpgradedNsxtClusterValidator" method="validate" line="80"] Checking for partially upgraded NSX clusters on domain: 4935846b-####-####-####-dfc9a6b354db
    2026-09-24T14:11:05.402Z DEBUG vcf_lcm 561923 [sddcm@4413 traceId="########" spanId="c01a" threadId="http-nio-127.0.0.1-7400-exec-6" class="c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil" method="isEdgeUpgradeAfterHost" line="5371"] Could not get NSX-T target version for domain 4935846b-####-####-####-dfc9a6b354db: Invalid domain release view. Plan for domain 4935846b-####-####-####-dfc9a6b354db does not exist.
    2026-09-24T14:11:05.403Z INFO vcf_lcm 561923 [sddcm@4413 traceId="########4" spanId="c01a" threadId="http-nio-127.0.0.1-7400-exec-6" class="c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil" method="isEdgeUpgradeAfterHost" line="5399"] Domain 4935846b-####-####-####-dfc9a6b354db has NSX-T inventory version <version>, isEdgeUpgradeAfterHost: false
    2026-09-24T14:11:05.403Z DEBUG vcf_lcm 561923 [sddcm@4413 traceId="########" spanId="c01a" threadId="http-nio-127.0.0.1-7400-exec-6" class="c.v.e.s.l.s.UpgradeServiceHelper" method="getPartiallyUpgradedNsxtClusters" line="1422"] NSX cluster with ID: <NSX VIP FQDN>, name: <NSX VIP FQDN>l, version: {"version":"<version>"} is partially upgraded
    2026-09-24T14:11:05.404Z ERROR vcf_lcm 561923 [sddcm@4413 traceId="########" spanId="c01a" threadId="http-nio-127.0.0.1-7400-exec-6" class="c.v.e.s.l.s.v.i.PartiallyUpgradedNsxtClusterValidator" method="validate" line="88"] Domain 4935846b-####-####-####-dfc9a6b354db with nsx cluster <NSX VIP FQDN> is partially upgraded.

Environment

  • VCF 9.1.1
  • NSX 4.x

Cause

An older NSX upgrade was never fully marked as completed. SDDC Manager sees that as an in-progress upgrade and will not allow a new upgrade to be started.

Resolution

Part 1: Manually upload the NSX 9.1.1 MUB file to NSX

  1. Download the target version of VCF NSX 9.x. MUB (Management Upgrade Bundle) from Broadcom downloads.
  2. Copy the MUB file to the /image directory on one of the NSX manager nodes.

  3. Log in as root user on that manager and run the following API to upload the MUB:

    curl -k -u admin -X POST "https://localhost/api/v1/upgrade/bundles?action=upload" -H "Content-Type: multipart/form-data" -F "file=@/image/VMware-NSX-upgrade-bundle-<version>.mub"

  4. Check the upload status, this may take some time:

    curl -k -u admin -X GET 'https://localhost/api/v1/upgrade/bundles/latest/status'

    Note: Poll the upload status with above API until the status is SUCCESS, percentage 50 and step is UPLOAD_COMPLETE

  5. Accept EULA

    curl -k -u admin -X POST 'https://localhost/api/v1/upgrade/eula/accept'

  6. Upgrade UC (Upgrade coordinator) using the API:

    curl -k -u admin -X POST 'https://localhost/api/v1/upgrade?action=upgrade_uc'

  7. Check UC upgrade status, this may take some time:

    curl -k -u admin -X GET 'https://localhost/api/v1/upgrade/uc-upgrade-status'

    Note: Poll until the state is SUCCESS, progress_percentage is 100 and status is "Upgrade-coordinator has been upgraded".


Part 2: Configure SDDC Manager to allow the NSX upgrade

  1. Take a snapshot of the SDDC Manager VM.
  2. Log in to the SDDC Manager UI and navigate to Inventory > Workload Domains > <problematic workload domain>.
  3. The URL bar of your browser should have a portion that looks similar to the following:

    /ui/sddc-manager/inventory/domains/mgmt-vi-domains/<UUID>/summary

    Make a note of the UUID value.
  4. SSH to the SDDC Manager VM as the vcf user.
  5. Issue the following commands to generate an authorization token:

    PASSWORD=$(systemd-ask-password "Enter [email protected] password: ")
    PAYLOAD=$(jq -n --arg user "[email protected]" --arg pwd "$PASSWORD" '{username: $user, password: $pwd}')
    TOKEN=$(curl -sd "$PAYLOAD" -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken'

  6. Issue the echo $TOKEN command to validate that the TOKEN variable is populated.
  7. Issue the following command to see the release information for the workload domain:

    curl -sk -H "Authorization: Bearer $TOKEN" 'https://localhost/v1/releases/domains/<UUID from Step 2>' |jq |tee releases.json

    Note: A file named releases.json will be created and output similar to the following will be displayed:

     {
      "domainId": "<UUID from Step 2>",
      "targetVersion": "",
      "targetVxRailVersion": "",
      "targetSddcManagerVersion": "",
      "patches": [],
      "isCustomizedBom": false
    }

  8. Modify the releases.json file such that it looks like the following:

    {
      "targetVersion": "<version, i.e. 9.1.1.0>",
      "skipValidations": ["PARTIALLY_UPGRADED_NSXTCLUSTER_VALIDATOR"]
    }

  9. Issue the following command to apply the content of the modified releases.json file:

    curl -sk -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -X PATCH 'https://localhost/v1/releases/domains/4935846b-####-####-####-dfc9a6b354db' [email protected]

  10. In the VCF Operations UI, the ability to run the Prechecks and/or Upgrade NSX should be availalbe.
Powered by Wolken Software