Upgrading two workload domains sharing 1 NSX instance stuck at the NSX edge stage

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• The upgrade of the first workload domain (wld01) sharing an NSX instance with a second workload domain (wld02) gets stuck at the NSX edge stage because workload domain 2 is still running vSphere 9.0.2. Concurrently, the upgrade workflow for workload domain 2 is blocked.
  
  

• The following error pattern appears in lcm/lcm.log-debug:

  YYYY-MM-DDThh:mm:ss.368+0000 DEBUG [vcf_lcm,c17aa7bf5e7e46c1,bf82] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,http-nio-127.0.0.1-7400-exec-2] isFinalizeUpgradeNotComplete is true for NSX cluster <REDACTED_HOSTNAME> with finalizeUpgradeStatus as NOT_STARTED and isTargetVersionMatching true
YYYY-MM-DDThh:mm:ss.368+0000 DEBUG [vcf_lcm,c17aa7bf5e7e46c1,bf82] [c.v.e.s.l.s.UpgradeServiceHelper,http-nio-127.0.0.1-7400-exec-2] NSX cluster with ID: <REDACTED_HOSTNAME>, name: <REDACTED_HOSTNAME>, version: {"version":"9.0.2.0.25150386"} is partially upgraded as One of the Edge cluster is upgraded but finalize is pending

Environment

VMware Cloud Foundation 9.1

Cause

NSX host transport node upgrade is paused because VCF 9.1 requires vCenter and ESXi across all workload domains sharing the NSX instance to be upgraded to the target version before host transport nodes can proceed. Workload domain 2 (wld02) vCenter and ESXi hosts remain on 9.0.2, while upgrade plan modifications for wld01 are disallowed for domains associated with a partially upgraded NSX cluster, creating an upgrade deadlock.

Resolution

 

1. Attempt a component-level upgrade directly via the SDDC Manager UI to upgrade wld02's vCenter and ESXi without triggering the blocked domain upgrade planning workflow:

   • Log into the SDDC Manager UI and navigate to wld02.

   • Use the component upgrade option to directly upgrade the following components:

     • vCenter → 9.1.0.0100.25417926

     • ESXi hosts → 9.1.0.0100.25433460

   • Once wld02's vCenter and ESXi reach version 9.1, the NSX host transport node upgrade and finalization will proceed normally for both wld01 and wld02.


2. If the component-level upgrade option is also blocked, execute a PATCH API call to force-correct wld02's upgrade plan:

   • Navigate to SDDC Manager > Developer Center > TargetUpgradableVersion.

   • Execute API endpoint: PATCH /v1/releases/domains/e19756b4-a783-448a-a063-cca1e331cec5

   • Provide JSON payload:
{
  "targetVersion": "9.1.0.0",
  "patches": [
    {
      "productType": "VCENTER",
      "version": "9.1.0.0100.25417926"
    },
    {
      "productType": "HOST",
      "version": "9.1.0.0100.25433460"
    },
    {
      "productType": "NSX_T_MANAGER",
      "version": "9.1.0.0100.25470810"
    }
  ],
  "skipValidations": [
    "PARTIALLY_UPGRADED_NSXTCLUSTER_VALIDATOR"
  ]
}

 

Following the successful upgrade of wld02's vCenter and ESXi hosts to version 9.1, proceed with finalization of the NSX host transport node upgrade for both wld01 and wld02.

Additional Information

Upgrading vCenter and NSX Manager