NSX upgrade via SDDC manager completes on the NSX-T, but shows failed in the SDDC manager.

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x

Issue/Introduction

• NSX is being upgraded through SDDC manager, and shows failed.
• If you re-try the upgrade from the SDDC manager you see a message similar to the following:
  
  

  Description	Upgrade - NSX_T_UC nsxt.example.com:UpgradeCoordinator
  Progress Messages	com.vmware.vcf.error.runtime.nsxt.already.upgraded
  Error	Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded Remediation Message: NSX cluster is already upgraded. Download a new bundle (if available). Retry the upgrade, once available. Reference Token: KA3RC4

• If logged into the NSX Manager UI, you can see the upgrade has completed successfully on the System > Lifecycle Management > Upgrade page
  
  

Environment

VMware Cloud Foundation 5.x
VMware NSX 4.x

Cause

SDDC manager fails to update the NSX version in the Database, making the upgrade not completed. When re-trying it validates that the NSX packages are already installed.

To validate the version you can run the below command -

psql -h localhost -U postgres -d platform -c "select id, version, cluster_fqdn from nsxt where cluster_fqdn='<clusterFQDN>';"

Resolution

1. Take a snapshot of the SDDC manager VM.
2. Run the following command to update the version using the cluster_fqdn for the NSX-T to be corrected.
   
   # psql -h localhost -U postgres -d platform -c "update nsxt set version='<version>' where cluster_fqdn='<clusterFQDN>';"
   
   
3. Validate the change has been performed
   
   # psql -h localhost -U postgres -d platform -c "select id, version from nsxt where cluster_fqdn='clusterFQDN';"
   
   
4. Restart the LCM service on the SDDC manager.
   
   # systemctl restart lcm 

If you believe you have encountered this issue and the previous steps do not help to resolve the issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

Additional Information

A similar issue is observed in VCF 5.2 wherein the NSX upgrade via SDDC manager completes in NSX, but shows failed in the SDDC manager.

When the upgrade is retried, it fails with com.vmware.vcf.error.runtime.nsxt.already.upgraded.

Upgrade - NSX_T_UC :UpgradeCoordinator
Failed
Beginning of Expandable row content Screen reader table commands may not work for viewing expanded content, please use your screen reader's browse mode to read the content exposed by this button
Description    Upgrade - NSX_T_UC example.com:UpgradeCoordinator
Progress Messages    com.vmware.vcf.error.runtime.nsxt.already.upgraded.

See NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK for information on resolving this issue.