NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x

Issue/Introduction

• NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK.
• NSX shows upgraded to version 4.2 but SDDC UI does not take it.
• Error in SDDC Manager UI

  Upgrade - NSX_T_UC :UpgradeCoordinator
  Failed
  Beginning of Expandable row content Screen reader table commands may not work for viewing expanded content, please use your screen reader's browse mode to read the content exposed by this button
  Description    Upgrade - NSX_T_UC example.com:UpgradeCoordinator
  Progress Messages    com.vmware.vcf.error.runtime.nsxt.already.upgraded.

  Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded
  Remediation Message: NSX cluster is already upgraded. Download a new bundle (if available). Retry the upgrade, once available.

• Error in /var/log/vmware/vcf/lcm/lcm-debug.log

  vcf_lcm,0000000000000000,0000,upgradeId=xxxx,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=af-mgt-nsx01.ad.hs1int.com:_ParallelClusterUpgradeElement,bundleElementId=xxxx] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-2] Setting Upgrade Error for stage NSX_UPGRADE_STAGE_EDGE_POSTCHECK, error description Check overall transport node status: [Overall status of the edge transport node xxxx is DOWN.]:, remediation Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.
  
  DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=xxxx,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId:_ParallelClusterUpgradeElement,bundleElementId=xxxx] [c.v.e.s.l.d.c.u.UpgradeClientImpl,Upgrade-2] bundleSoftwareType = NSX_T_EDGE
  
  ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxx ,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId:_ParallelClusterUpgradeElement,bundleElementId=yyyy [c.v.e.s.l.p.i.n.s.NsxtEdgeClusterParallelUpgradeStageRunner,Upgrade-2] upgrade error for resource  { "errorType": "RECOVERABLE", "stage": "NSX_UPGRADE_STAGE_EDGE_POSTCHECK", "errorCode": "com.vmware.vcf.error.runtime.nsxt.edge.cluster.postcheck.failed", "errorDescription": "Check overall transport node status: [Overall status of the edge transport node xxxx is DOWN.]: , "metadata": "Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.", "metadataAttributes": { "LCM_LOG_LOCATION": "/
  var/log/vmware/vcf/lcm", "LCM_HOST_ADDRESS": "x.x.x.x" }, "referenceToken": "ABC" }

 

Environment

VMware Cloud Foundation 5.2

 

Cause

SDDC UI is not taking the target version of NSX.

Resolution

Update SDDC Manager Inventory with the updated version of NSX-T Manager.

Follow the below steps to update the SDDC Manager inventory for all the upgraded NSX-T instances

1. Take a snapshot before proceeding further.
2. SSH to SDDC Manager with vcf user and su to root
3. Get VCF deployed NSX-T Cluster Inventory Ids associated with domains.

   curl -v -k http://127.0.0.1:7100/inventory/nsxt | json_pp

   
   Sample output

    {
   
         "clusterIpAddress" : "x.x.x.x",
         "shared" : false,
         "status" : "ACTIVE",
         "version" : "<current NSX-T Version>",
         "domainIds" : [
            "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxc33d"
         ],
         "clusterFqdn" : "vip-nsx-mgmt.example.com",
         "id" : "<nsxt-entity-id>",
         "nsxtClusterDetails" : [
            {
               "ipAddress" : "x.x.x.x",
               "vmName" : "nsx-mgmt-1",
               "fqdn" : "nsx-mgmt-1.example.com",
               "id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1b90"
            }
         ],

4. At the bottom of the output, please note the <nsxt-entity-id> for the upgraded NSX-T Manager instance
5. For each NSX-T entity Id (NSX-T cluster id), update the NSX-T version with the correct version (Make sure you have already upgraded this NSX-T by logging to NSX-T cluster IP).NOTE: We do not need to update the IDs in the cluster details.

   curl -v -k http://127.0.0.1:7100/inventory/entities/<nsxt-entity-id> -X PATCH -d '{"type":"NSXT_CLUSTER","status":"ACTIVE","version":"<correct version>"}' -H 'Content-Type:application/json'