NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK
search cancel

NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK

book

Article ID: 379200

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation 5.x

Issue/Introduction

  • NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK.
  • NSX shows upgraded to version 4.2 but SDDC UI does not take it.
  • Error in SDDC Manager UI

    Description
    Progress Messages

    Upgrade - NSX_T_UC :UpgradeCoordinator
    Failed
    Beginning of Expandable row content Screen reader table commands may not work for viewing expanded content, please use your screen
    reader's browse mode to read the content exposed by this button
    Upgrade - NSX_T_UC example.com: UpgradeCoordinator

        Message: com. vmware.vcf.error.runtime.nsxt.already. upgraded. Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded
        Remediation Message: NSX cluster is already upgraded. Download a new bundle (if available). Retry the upgrade, once available.

  • Error in /var/log/vmware/vcf/lcm/lcm-debug.log

vcf_lcm,0000000000000000,0000,upgradeId=xxxx,resourceType=NSX_T_PARALLEL_CLUSTER, resourceId=af-mgt-
NSX_FQDN :_ ParallelClusterUpgradeElement, bundleElementId=xxxx] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil, Upgrade-2] Setting
Upgrade Error for stage NSX_UPGRADE_STAGE_EDGE_POSTCHECK, error description Check overall transport node status: [Overall status of
the edge transport node xxxx is DOWN. ]:, remediation Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and
address those errors. Please run the upgrade precheck and restart the upgrade.

DEBUG
[vcf_lcm,0000000000000000,0000,upgradeId=xxxx, resourceType=NSX_T_PARALLEL_CLUSTER, resourceId :_ ParallelClusterUpgradeElement, bundleEleme
[c.v.e.s.l.d.c.u.UpgradeClientImpl, Upgrade-2] bundleSoftwareType = NSX_T_EDGE

ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxx
,resourceType=NSX_T_PARALLEL_CLUSTER, resourceId :_ ParallelClusterUpgradeElement, bundleElementId=yyyy
[c.v.e.s.l.p.i.n.s.NsxtEdgeClusterParallelUpgradeStageRunner, Upgrade-2] upgrade error for resource { "errorType": "RECOVERABLE",
"stage": "NSX_UPGRADE_STAGE_EDGE_POSTCHECK", "errorCode": "com. vmware. vcf.error.runtime.nsxt.edge. cluster.postcheck. failed",
"errorDescription": "Check overall transport node status: [Overall status of the edge transport node xxxx is DOWN. ]: , "metadata":
"Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck
and restart the upgrade.", "metadataAttributes": { "LCM_LOG_LOCATION": "/
var/log/vmware/vcf/lcm", "LCM_HOST_ADDRESS": "x.x.x.x" }, "referenceToken": "ABC" }

 

Environment

VMware Cloud Foundation 5.2

 

Cause

SDDC UI is not taking the target version of NSX.

Resolution

Update SDDC Manager Inventory with the updated version of NSX-T Manager.


Follow the below steps to update the SDDC Manager inventory for all the upgraded NSX-T instances

  1. Take a snapshot before proceeding further.
  2. SSH to SDDC Manager with vcf user and su to root
  3. Get VCF deployed NSX-T Cluster Inventory Ids associated with domains.

          curl -v -k http://127.0.0.1:7100/inventory/nsxt | json_pp

        Sample output

{

"clusterIpAddress" : "x.x.x.x",
"shared" : false,
"status" : "ACTIVE",
"version" : "<current NSX-T Version>",
"domainIds" : [
"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxc33d"

"clusterFqdn" : "vip-nsx-mgmt.example.com",
"id" : "<nsxt-entity-id>",
"nsxtClusterDetails" : [

"ipAddress" : "x.x.x.x",
"vmName" : "nsx-mgmt-1",
"fqdn" : "nsx-mgmt-1.example.com",
"id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1b90"

}

         

       Sample output

         {

"clusterIpAddress" : "x.x.x.x"
"shared" : false,
"status" : "ACTIVE",
"version" : "<current NSX-T Version>",
"domainIds" : [
"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxc33d"

"clusterFqdn" : "vip-nsx-mgmt.example.com",
"id" : "<nsxt-entity-id>",
"nsxtClusterDetails" : [

"ipAddress" : "x.x.x.x",
"vmName" : "nsx-mgmt-1",
"fqdn" : "nsx-mgmt-1.example.com",
"id

}

xx-xxxxxxxx1b90"

{

"clusterIpAddress" :
"shared" : false,
"status" : "ACTIVE",
"version" : "<current NSX-T Version>",
"domainIds" : [
"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxc33d"

"clusterFqdn" : "vip-nsx-mgmt.example.com",
"id" : "<nsxt-entity-id>",
"nsxtClusterDetails" : [

"ipAddress" : "x.x.x.x",
"vmName" : "nsx-mgmt-1",
"fqdn" : "nsx-mgmt-1.example.com",
"id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1b90"

"x.x.x.x"

           },

 

    }

{

"clusterIpAddress" :
"shared" : false,
"status" : "ACTIVE",
"version" : "<current NSX-T Version>",
"domainIds" : [
"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxc33d"

"clusterFqdn" : "vip-nsx-mgmt.example.com",
"id" : "<nsxt-entity-id>",
"nsxtClusterDetails" : [

"ipAddress" : "x.x.x.x",
"vmName" : "nsx-mgmt-1",
"fqdn" : "nsx-mgmt-1.example.com",
"id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1b90"

"x.x.x.x"

},

         },

  1. At the bottom of the output, please note the <nsxt-entity-id> for the upgraded NSX-T Manager instance
  2. For each NSX-T entity Id (NSX-T cluster id), update the NSX-T version with the correct version (Make sure you have already upgraded this NSX-T by logging to NSX-T cluster IP).NOTE: We do not need to update the IDs in the cluster details.

     

    curl -v -k http://127.0.0.1:7100/inventory/entities/<nsxt-entity-id> -X PATCH -d
    '{"type": "NSXT_CLUSTER","status": "ACTIVE","version":"<correct version>"}' -H 'Content-Type: application/json'
Powered by Wolken Software