NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK
search cancel

NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK

book

Article ID: 379200

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation 5.x

Issue/Introduction

  • NSX-T Upgrade from SDDC manager fails at NSX_UPGRADE_STAGE_CLUSTER_PRECHECK.

  • NSX shows upgraded to version 4.2 but SDDC UI does not reflect it.

  • Error in SDDC Manager UI, similar to the below:

    Description: 

    Upgrade - NSX_T_UC :UpgradeCoordinator 

    Failed

    Beginning of Expandable row content Screen reader table commands may not work for viewing expanded content, please use your screen
    reader's browse mode to read the content exposed by this button
    Upgrade - NSX_T_UC example.com: UpgradeCoordinator

    Message: com. vmware.vcf.error.runtime.nsxt.already. upgraded. Message: com.vmware.vcf.error.runtime.nsxt.already.upgraded 

    Remediation Message: NSX cluster is already upgraded. Download a new bundle (if available). Retry the upgrade, once available.

  • Error in /var/log/vmware/vcf/lcm/lcm-debug.log similar to: 

    vcf_lcm,0000000000000000,0000,upgradeId=xxxx,resourceType=NSX_T_PARALLEL_CLUSTER, resourceId=af-mgt-
    NSX_FQDN :_ ParallelClusterUpgradeElement, bundleElementId=xxxx] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil, Upgrade-2] Setting
    Upgrade Error for stage NSX_UPGRADE_STAGE_EDGE_POSTCHECK, error description Check overall transport node status: [Overall status of
    the edge transport node xxxx is DOWN. ]:, remediation Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and
    address those errors. Please run the upgrade precheck and restart the upgrade.

    DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=xxxx, resourceType=NSX_T_PARALLEL_CLUSTER, resourceId :_ ParallelClusterUpgradeElement, bundleEleme
    [c.v.e.s.l.d.c.u.UpgradeClientImpl, Upgrade-2] bundleSoftwareType = NSX_T_EDGE

    ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxx ,resourceType=NSX_T_PARALLEL_CLUSTER, resourceId :_ ParallelClusterUpgradeElement, bundleElementId=yyyy
    [c.v.e.s.l.p.i.n.s.NsxtEdgeClusterParallelUpgradeStageRunner, Upgrade-2] upgrade error for resource { "errorType": "RECOVERABLE",
    "stage": "NSX_UPGRADE_STAGE_EDGE_POSTCHECK", "errorCode": "com. vmware. vcf.error.runtime.nsxt.edge. cluster.postcheck. failed",
    "errorDescription": "Check overall transport node status: [Overall status of the edge transport node xxxx is DOWN. ]: , "metadata":
    "Check for errors in the LCM log files at x.x.x.x:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck
    and restart the upgrade.", "metadataAttributes": { "LCM_LOG_LOCATION": "/var/log/vmware/vcf/lcm", "LCM_HOST_ADDRESS": "x.x.x.x" }, "referenceToken": "ABC" }

 

Environment

VMware Cloud Foundation 5.2

VMware NSX

 

Cause

This issue occures when the SDDC Manager fails to insert the updated target version of NSX.

Resolution

This is a condition that may occur in an SDDC environment. In order to workaround the issue, you will update the SDDC Manager Inventory with the correct version of NSX-T Manager.

Workaround:

Follow the below steps to update the SDDC Manager inventory for all the upgraded NSX-T instances

  1. Take a snapshot before proceeding further.
  2. SSH to SDDC Manager with vcf user and su to root
  3. Get VCF deployed NSX-T Cluster Inventory Ids associated with domains. 
    # curl -v -k http://127.0.0.1:7100/inventory/nsxt | json_pp

    Sample output: 

    {
    "clusterIpAddress" : "x.x.x.x",
    "shared" : false,
    "status" : "ACTIVE",
    "version" : "<current NSX-T Version>",
    "domainIds" : [
    "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxc33d"
    "clusterFqdn" : "vip-nsx-mgmt.example.com",
    "id" : "<nsxt-entity-id>",
    "nsxtClusterDetails" : [
    "ipAddress" : "x.x.x.x",
    "vmName" : "nsx-mgmt-1",
    "fqdn" : "nsx-mgmt-1.example.com",
    "id" : "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1b90"
    }

  4. At the bottom of the output, please note the <nsxt-entity-id> for the upgraded NSX-T Manager instance

  5. For each NSX-T entity Id (NSX-T cluster id), update the NSX-T version with the correct version (Make sure you have already upgraded this NSX-T by logging to NSX-T cluster IP).

    curl -v -k http://127.0.0.1:7100/inventory/entities/<nsxt-entity-id> -X PATCH -d '{"type": "NSXT_CLUSTER","status": "ACTIVE","version":"<correct version>"}' -H 'Content-Type: application/json'

    NOTE: Do not update any IDs in the cluster details.
Powered by Wolken Software