NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK
search cancel

NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK

book

Article ID: 379200

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation 5.x

Issue/Introduction

  • NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK.
  • NSX shows upgraded to version 4.2 but SDDC UI does not take it.
  • Error in SDDC Manager UI
    Upgrade - NSX_T_UC :UpgradeCoordinator
    Failed
    Beginning of Expandable row content Screen reader table commands may not work for viewing expanded content, please use your screen reader's browse mode to read the content exposed by this button
    Description    Upgrade - NSX_T_UC example.com:UpgradeCoordinator
    Progress Messages    com.vmware.vcf.error.runtime.nsxt.already.upgraded.
    Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded
    Remediation Message: NSX cluster is already upgraded. Download a new bundle (if available). Retry the upgrade, once available.
  • Error in /var/log/vmware/vcf/lcm/lcm-debug.log
    vcf_lcm,0000000000000000,0000,upgradeId=xxxx,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=af-mgt-nsx01.ad.hs1int.com:_ParallelClusterUpgradeElement,bundleElementId=xxxx] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-2] Setting Upgrade Error for stage NSX_UPGRADE_STAGE_EDGE_POSTCHECK, error description Check overall transport node status: [Overall status of the edge transport node xxxx is DOWN.]:, remediation Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.
    
    DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=xxxx,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId:_ParallelClusterUpgradeElement,bundleElementId=xxxx] [c.v.e.s.l.d.c.u.UpgradeClientImpl,Upgrade-2] bundleSoftwareType = NSX_T_EDGE
    
    ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxx ,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId:_ParallelClusterUpgradeElement,bundleElementId=yyyy [c.v.e.s.l.p.i.n.s.NsxtEdgeClusterParallelUpgradeStageRunner,Upgrade-2] upgrade error for resource  { "errorType": "RECOVERABLE", "stage": "NSX_UPGRADE_STAGE_EDGE_POSTCHECK", "errorCode": "com.vmware.vcf.error.runtime.nsxt.edge.cluster.postcheck.failed", "errorDescription": "Check overall transport node status: [Overall status of the edge transport node xxxx is DOWN.]: , "metadata": "Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.", "metadataAttributes": { "LCM_LOG_LOCATION": "/
    var/log/vmware/vcf/lcm", "LCM_HOST_ADDRESS": "x.x.x.x" }, "referenceToken": "ABC" }

 

Environment

VMware Cloud Foundation 5.2

 

Cause

SDDC UI is not taking the target version of NSX.

Resolution

Update SDDC Manager Inventory with the updated version of NSX-T Manager.


Follow the below steps to update the SDDC Manager inventory for all the upgraded NSX-T instances

  1. Take a snapshot before proceeding further.
  2. SSH to SDDC Manager with vcf user and su to root
  3. Get VCF deployed NSX-T Cluster Inventory Ids associated with domains.
    curl -v -k http://127.0.0.1:7100/inventory/nsxt | json_pp


    Sample output

     {
    
          "clusterIpAddress" : "x.x.x.x",
          "shared" : false,
          "status" : "ACTIVE",
          "version" : "<current NSX-T Version>",
          "domainIds" : [
             "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxc33d"
          ],
          "clusterFqdn" : "vip-nsx-mgmt.example.com",
          "id" : "<nsxt-entity-id>",
          "nsxtClusterDetails" : [
             {
                "ipAddress" : "x.x.x.x",
                "vmName" : "nsx-mgmt-1",
                "fqdn" : "nsx-mgmt-1.example.com",
                "id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1b90"
             }
          ],
  4. At the bottom of the output, please note the <nsxt-entity-id> for the upgraded NSX-T Manager instance
  5. For each NSX-T entity Id (NSX-T cluster id), update the NSX-T version with the correct version (Make sure you have already upgraded this NSX-T by logging to NSX-T cluster IP).NOTE: We do not need to update the IDs in the cluster details.
    curl -v -k http://127.0.0.1:7100/inventory/entities/<nsxt-entity-id> -X PATCH -d '{"type":"NSXT_CLUSTER","status":"ACTIVE","version":"<correct version>"}' -H 'Content-Type:application/json'