"NSX UC Rollback failed as UC is not upgraded" precheck error for VCF 9 upgrade

Products

• VMware SDDC Manager
• VMware NSX

Issue/Introduction

• On VCF/SDDC UI, NSX upgrade pre-checks fail with the following error

Description: Initiate NSX UC Rollback after receiving the precheck response
Error Description: NSX UC Rollback failed as UC is not upgraded
Impact: High: NSX UC is not rolled back as UC is not upgraded.
Remediation Please contact GSS for assistance. Retry the precheck after fixing the issues (if any)."

• SDDC Manager logs may have an error similar to this example

Message: com.vmware.vcf.error.runtime.nsxt.payload.upload.failed. NSX Upgrade Coordinator - Failed to upload NSX bundle. Review the LCM log files on the SDDC Manager appliance at 127.0.0.1:/var/log/vmware/vcf/lcm and resolve identified errors.

Environment

VMware SDDC Manager 9.0
VMware NSX 9.0

Cause

The VCF Ops runs prechecks, then rolls them back, but NSX does not perform the same, creating an inconsistency in the upgrade workflow. 

Resolution

This is a known issue impacting VCF 9.x.

To workaround this issue, go to the NSX UI and initiate the NSX upgrade from there instead.

1. Upload the .pub file and run prechecks
2. Upload the .mub file and perform the upgrade
3. VCF Ops performs an inventory sync detecting the upgraded NSX instance
4. Continue VCF upgrade workflow from VCF Ops

Additional Information

NSX 9.0 Release Notes