NSX-T upgrade from SDDC Manager is failing at NSX_UPGRADE_STAGE_EDGE_PRECHECK stage

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x

Issue/Introduction

• NSX-T upgrade from SDDC Manager is failing at NSX_UPGRADE_STAGE_EDGE_PRECHECK stage
• NSX-T UI > Upgrade tab has the below error

  Pre-upgrade checks failed for EDGE. Failed to execute check for NSX Application Platform version for upgrade. The system encountered a problem and is unable to connect the NSX Application Platform

• SDDC Manager's /var/log/vmware/vcf/lcm/lcm-debug.log has the below error

  [vcf_lcm,0000000000000000,0000,upgradeId=6b5#####-####-####-####-########23b4, resourceType=NSX_T_PARALLEL_CLUSTER, resourceId=nsx.example.com :_ ParallelClusterUpgradeElement, bundleElementId=aa3#####-####-####-####-########bcbb]
  [c.v.e.s.l.p.i.n.s.NsxtEdgeClusterParallelUpgradeStageRunner, Upgrade-2] upgrade error for resource nsx.example.com:467#####-####-####-####-#######e5cf7 : { "errorType": "RECOVERABLE", "stage": "NSX_UPGRADE_STAGE_EDGE_PRECHECK",
  "errorCode": "com. vmware. vcf.error.runtime.nsxt.edge. cluster.precheck. failed", "errorDescription": "Check for open alarms on edge node .: [Edge node faf#####-####-####-####-########e37a has 3 open alarm(s) present. Kindly resolve the open alarm(s) before proceeding with the upgrade. ]: edge1, \nCheck for NSX Application Platform version for upgrade: [Failed to execute Check for NSX Application Platform version for upgrade. ]: edge1, edge2, \nCheck for open alarms on edge node .: [Edge node 4c3#####-####-####-####-########634e has 3 open alarm(s) present. Kindly resolve the open alarm(s) before proceeding with the upgrade. ]: edge2", "metadata": "Check for errors in the LCM log files at 127.0.0.1:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.", "metadataAttributes": { "LCM_LOG_LOCATION": "/var/log/vmware/vcf/lcm", "LCM_HOST_ADDRESS": "127.0.0.1" }, "referenceToken": "######"

• NSX-T UI has open alarms for the Edge nodes
• NSX Appliance Platform status is in degraded state and NSX Appliance Platform version is blank in NSX-T UI.

 

Cause

NAPP version is stored in an internal cache during startup. This version should remain persistent until NAPP is upgraded.  However, the upgrade pre-check fails to validate the version due to it missing in the cache. The cache had gotten cleaned up erroneously due to a cron job that runs once a week. 

This impacts only NSX upgrade operation if NAPP 4.2.0 is installed and upgrade is attempted at least a week after NAPP startup. The version will not be shown in the NAPP UI once the cache has been cleared, but there is no operational impact.

Resolution

• Resolve the issues reported in open alarms in NSXT UI - Refer Working with Events and Alarms
• Follow the steps in KB NSX Upgrade fails during pre-check stage due to failure in getting NAPP version to resolve the issue with NSX Appliance Platform version in NSXT UI
• Retry the NSX-T upgrade from SDDC Manager.