Unable to proceed with domain upgrade on the SDDC Manager/VCF-Ops due to NSX in partial upgrade state.

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• After upgrading the management domain, you are trying to upgrade the workload domain on the SDDC manager.
  
  
• The Plan upgrade option is available, however, once you click on plan upgrade and select the VCF version you need to upgrade to - there is an error without any message displayed and the Confirm option is grayed out.

• Taking the Domain ID from the browser URL, you check the errors in /var/log/vmware/vcf/lcm/lcm-debug.log. You see the below error trace where it mentions that NSX-T is partially upgraded: 

YYYY-MM-DDT12:33:48.178+0000 DEBUG [vcf_lcm,54b6599ba21d4ff4,66fe] [c.v.e.s.l.s.v.i.ValidationServiceImpl,http-nio-127.0.0.1-7400-exec-6] Validation errors collected: [Domain xxxx-xxxx-xxxx-xxxx with nsx cluster nsx_fqdn is partially upgraded.]

• In the Plan Patching section, you will see the below error as well:

• In the NSX 9.0.1 UI > System > Lifecycle Management > Upgrade, the UPGRADE SUMMARY > Prepare for Uprgade is presenting a warning saying "This NSX Manager is managed by VCF Operations. Upgrading NSX through this workflow may impact VCF Operations."
  
  
• In 9.x Plan Domain Upgrade or Plan Component Upgrade,  the wizard blocks because of error - "Domain xxxx-xxxx-xxxx-xxxx with nsx cluster nsx_fqdn is partially upgraded."
  
  
  
  

 

Environment

VMware Cloud Foundation 5.x

VMware Cloud Foundation 9.x

NSX Manager 9.x

Cause

• A pending/paused upgrade task on the NSX Manager connected to the SDDC manager is ongoing due to which the SDDC Manger is unable to proceed with the upgrade 

Resolution

In pre-9 version NSX manager UI:

• Login to NSX manager UI > system > upgrades to confirm there is an upgrade pending/paused:

• Proceed with the NSX upgrade and you will be able to upgrade the SDDC manager.
  
  

In 9.x NSX manager UI:

• Login to NSX manager UI > System > Lifecycle Management > Upgrade, click on CONTINUE WITH UPGRADE to finish upgrade NSX manager.
  
  
• Proceed with upgrading other components.