NSX-T upgrade via SDDC manager completes on the NSX-T, but shows failed in the SDDC manager.

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x

Issue/Introduction

• NSX-T is being upgraded through SDDC manager, and shows failed.
• If you re-try the upgrade from the SDDC manager again it gives the below error message - 

Description	Upgrade - NSX_T_UC nsxt.example.com:UpgradeCoordinator
Progress Messages	com.vmware.vcf.error.runtime.nsxt.already.upgraded
Error	Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded Remediation Message: NSX cluster is already upgraded. Download a new bundle (if available). Retry the upgrade, once available. Reference Token: KA3RC4

• However, if logged into the NSX-T manager you can see the upgrade has been completed by navigating to System->Upgrade

Environment

VMware Cloud Foundation 5.2

Cause

SDDC manager fails to update the NSX version on the Database, making the upgrade not completed. When re-trying it validates that the NSX packages are already installed.

To validate the version you can run the below command -

# /usr/pgsql/13/bin/psql -h localhost -U postgres
# \c platform;
# select id, status, version, cluster_fqdn from nsxt;

Resolution

1. Take a snapshot of the SDDC manager VM.

2. Connect to postgres DB and then connect to platform DB

# /usr/pgsql/13/bin/psql -h localhost -U postgres
# \c platform;

3. Run the below command to update the version using the cluster_fqdn for the NSX-T to be corrected.

# update nsxt set version='4.2.0.0.0-24105817' where cluster_fqdn='nsxt.example.com';

4. Restart the LCM service on the SDDC manager.

# systemctl restart lcm 

 

Additional Information

Similar issue is observed in VCF 5.2 wherein NSX-T upgrade via SDDC manager completes on the NSX-T, but shows failed in the SDDC manager.

When we re-try the upgrade, it fails with com.vmware.vcf.error.runtime.nsxt.already.upgraded.

Upgrade - NSX_T_UC :UpgradeCoordinator
Failed
Beginning of Expandable row content Screen reader table commands may not work for viewing expanded content, please use your screen reader's browse mode to read the content exposed by this button
Description    Upgrade - NSX_T_UC example.com:UpgradeCoordinator
Progress Messages    com.vmware.vcf.error.runtime.nsxt.already.upgraded.

NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK