NSX upgrade via SDDC manager completes on the NSX, but shows failed in the SDDC manager.

Products

• VMware SDDC Manager
• VMware NSX

Issue/Introduction

• NSX upgrade task from SDDC manager shows failed.
• Re-trying the upgrade from the SDDC manager shows message similar to the following:
  
  

  Description	Upgrade - NSX_T_UC nsxt.example.com:UpgradeCoordinator
  Progress Messages	com.vmware.vcf.error.runtime.nsxt.already.upgraded
  Error	Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded Remediation Message: NSX cluster is already upgraded. Download a new bundle (if available). Retry the upgrade, once available. Reference Token: ######

• NSX Manager UI will show the upgrade has completed successfully on the System > Lifecycle Management > Upgrade page
  
  
  
  
• The following error can also been seen upon re-trying upgrade from SDDC manager and NSX upgrade fails with com.vmware.vcf.error.runtime.nsxt.already.upgraded.

Upgrade - NSX_T_UC :UpgradeCoordinator
Failed
Beginning of Expandable row content Screen reader table commands may not work for viewing expanded content, please use your screen reader's browse mode to read the content exposed by this button
Description    Upgrade - NSX_T_UC example.com:UpgradeCoordinator
Progress Messages    com.vmware.vcf.error.runtime.nsxt.already.upgraded.

Environment

• VCF 5.x
• NSX 4.x

Cause

SDDC manager fails to update the NSX version in the Database, making the upgrade not completed. When re-trying it validates that the NSX packages are already installed.

Resolution

Follow the below steps to update the SDDC Manager inventory for all the upgraded NSX instances

1.Take a snapshot of the SDDC manager before proceeding further.

2. SSH to SDDC Manager with vcf user and su to root

3. Get VCF deployed NSX Cluster Inventory Ids associated with domains.

# curl -v -k http://127.0.0.1:7100/inventory/nsxt | json_pp

Sample output: 

{
"clusterIpAddress" : "###.###.###.###",
"shared" : false,
"status" : "ACTIVE",
"version" : "<previous NSX Version>",
"domainIds" : [
"########-####-####-####-########c33d"
"clusterFqdn" : "vip-nsx-mgmt.example.com",
"id" : "<nsxt-entity-id>",
"nsxtClusterDetails" : [
"ipAddress" : "###.###.###.###",
"vmName" : "nsx-mgmt-1",
"fqdn" : "nsx-mgmt-1.example.com",
"id" : "########-####-####-####-########1b90"
}

4. At the bottom of the output, please note the <nsxt-entity-id> for the upgraded NSX Manager instance

5. For each NSX entity Id (NSX cluster id), update the NSX version with the correct version (Validated the NSX version is updated by logging to NSX cluster IP).

curl -v -k http://127.0.0.1:7100/inventory/entities/<nsxt-entity-id> -X PATCH -d '{"type": "NSXT_CLUSTER","status": "ACTIVE","version":"<updated NSX version>"}' -H 'Content-Type: application/json'

NOTE: Do not update any IDs in the cluster details.

 

If the above steps do not resolve the issue, additional troubleshooting would be required to update the NSX version in to SDDC Manager Database. Please open a ticket with Broadcom support for assistance.