Unable to proceed with domain upgrade on the SDDC Manager/VCF-Ops due to NSX in partial upgrade state.

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• After upgrading the management domain, an attempt to upgrade a workload domain via SDDC Manager may stall or fail.
• When selecting Plan Upgrade and choosing the target VCF version, an error occurs without a descriptive message, and the Confirm option remains grayed out.

• Extracting the Domain ID from the browser URL and checking /var/log/vmware/vcf/lcm/lcm-debug.log reveals the following error trace indicating that NSX is in a partially upgraded state:
  YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,#############,####] [c.v.e.s.l.s.v.i.ValidationServiceImpl,http-nio-127.0.0.1-7400-exec-6] Validation errors collected: [Domain ####-####-####-#### with nsx cluster nsx_fqdn is partially upgraded.]

• Under the Plan Patching section, following error is also noticed:

• Additionally, in the NSX 9.0.1 UI (System > Lifecycle Management > Upgrade), the UPGRADE SUMMARY > Prepare for Upgrade section displays the warning:
  "This NSX Manager is managed by VCF Operations. Upgrading NSX through this workflow may impact VCF Operations."
  
  
• On VCF Operations, Plan Domain Upgrade or Plan Component Upgrade wizard, the process is blocked by the error: - "Domain ####-####-####-#### with nsx cluster <NSX_FQDN> is partially upgraded."
  
  
  
  

 

Environment

VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x
NSX Manager 9.x

Cause

A paused or a pending upgrade task exists on the NSX Manager connected to SDDC Manager, preventing SDDC Manager from initiating or finalizing the domain upgrade workflow.

Resolution

To resolve this issue, the pending or paused NSX Manager upgrade must be completed directly from the NSX Manager UI. Once the NSX upgrade finishes, the workload domain upgrade in SDDC Manager can proceed.

In NSX 4.x / Pre-9.x Manager UI
To clear the partial upgrade state, verify the current upgrade status and manually resume the operation:

1. Log in to the NSX Manager UI and navigate to System > Upgrades to verify if an upgrade is pending or paused:
2. Resume and complete the NSX upgrade.
3. Re-attempt the upgrade process within SDDC Manager.
   
   

In 9.x NSX manager UI
To finalize the pending upgrade workflow and release the lock on SDDC Manager, continue the upgrade from the Lifecycle Management page

1. Log in to the NSX Manager UI and navigate to System > Lifecycle Management > Upgrade.
2. Click CONTINUE WITH UPGRADE to complete the NSX Manager upgrade workflow.
   
3. Once the NSX Manager is upgraded proceed with upgrading the remaining workload domain components in SDDC Manager.