Error: com.vmware.vcf.error.runtime.nsxt.already.upgraded during VCF upgrade

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• During a workload domain upgrade from VCF 5.x to 9.x, the NSX-T upgrade sequence becomes stuck in the SDDC Manager UI.
• The SDDC Manager UI reflects a Running or Failed status for NSX-T, while the NSX Manager UI shows the upgrade has completed successfully.
• Retrying the upgrade task fails with the error: com.vmware.vcf.error.runtime.nsxt.already.upgraded.
• The lcm.log file on the SDDC Manager contains entries similar to: Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded

Environment

• VMware Cloud Foundation (VCF) 5.x, 9.x
• VMware NSX 4.x, 9.x
• VMware SDDC Manager

Cause

This issue occurs due to a synchronization failure where the SDDC Manager inventory is not updated after the NSX-T Upgrade Coordinator finishes successfully. Because the SDDC Manager database still lists the previous version or shows an incomplete task, the upgrade workflow cannot progress, and manual reconciliation is required.

Resolution

Fixed in VCF 9.x and higher. See Download Broadcom products and software for steps to download this release.
For instances where the inventory remains out of sync after a successful out-of-band upgrade, follow these steps to reconcile the state:
1. Verify NSX Status: Log in to the NSX Manager UI and navigate to System > Lifecycle Management > Upgrade to confirm all components show as Successful.
2. Snapshot SDDC Manager: Take a virtual machine snapshot of the SDDC Manager appliance before proceeding.
3. Update Inventory via API:
• Log in to the SDDC Manager via SSH as the vcf user and su to root.
• Retrieve the NSX Cluster ID: curl -v -k http://127.0.0.1:7100/inventory/nsxt | json_pp
• Update the inventory version using the Cluster ID (####) and target version (VERSION): curl -v -k http://127.0.0.1:7100/inventory/entities/#### -X PATCH -d '{"type": "NSXT_CLUSTER","status": "ACTIVE","version":"VERSION"}' -H 'Content-Type: application/json'
4. Reconcile Database Task (if UI is still stuck):
• Access the lcm database: psql -h localhost -U postgres -d lcm
• Locate the stuck task in the upgrade_activity_log and upgrade_element tables.
• Update the status from COMPLETED_WITH_FAILURE or RUNNING to COMPLETED_WITH_SUCCESS.
5. Restart Services:
• Exit the database and run: systemctl restart lcm
To speak with a customer representative or a Support Engineer see Contact Support - Broadcom Software. Scroll to the bottom of the page and click on your respective region.

Additional Information

• Refer to Article 378117 for detailed curl command syntax for inventory updates.
• Refer to Article 418202 for specific PostgreSQL reconciliation queries.