NSX-T Upgrade from SDDC Manager fails as it SKIPPED upgrade for all components

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x

Issue/Introduction

• NSX-T upgrade from SDDC Manager fails immediately.
• NSX Upgrade Coordinator, NSX Host Cluster, NSX Edge upgrades are marked as SKIPPED in SDDC Manager UI.
• SDDC Manager lcm-debug.log (/var/log/vmware/vcf/lcm/lcm-debug.log) shows similar to the below error messages

  [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxe1ce7,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=vip-nsx.example.com:_ParallelClusterUpgradeElement,bundleElementId=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyd783] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-2] Failed to calculate NSX upgrade compatibility
  com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to find products current version for NSX Advanced Load Balancer with vcfId {0}. Please verify if the NSX Manager is reachable.

  [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxe1ce7,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=vip-nsx.example.com:_ParallelClusterUpgradeElement,bundleElementId=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyd783] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-2] Handling NSX Exception
  com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to find products current version for NSX Advanced Load Balancer with vcfId {0}. Please verify if the NSX Manager is reachable.

  [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1ce7,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=vip-nsx.example.com:_ParallelClusterUpgradeElement,bundleElementId=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyd783] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-2] UpgradeError while calculating VVS compatibility, stage: NSX_UPGRADE_STAGE_MANAGER_PRECHECK, errorType: RECOVERABLE,errorCode: com.vmware.vcf.error.runtime.nsx.vvs.upgrade.compatibility.failed, errorDescription: Upgrade is not VVS compatible to continue with null error_message : Failed to find products current version for NSX Advanced Load Balancer with vcfId {0}. Please verify if the NSX Manager is reachable., httpStatus : , error_code : 0 Upgrade and interop incompatible product pair details are provided below(if any)
  :[] Products and Versions that have no compatibility data include(if any) :[], metadata NSX Upgrade compatibility checks failed. NSX upgrade bundle is not compatible with your existing version. Refer to the https://kb.vmware.com/s/article/91728 to address incompatibilities and proceed with the upgrade., metadataCode null

  [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1ce7,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=vip-nsx.example.com:_ParallelClusterUpgradeEl
  ement,bundleElementId=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyd783] [c.v.e.s.l.d.c.u.UpgradeClientImpl,Upgrade-2] Error occurred when updating UpgradeElement status
  java.lang.RuntimeException: Cannot modify the upgrade status of the upgrade in the current state QUEUED

Environment

VMware Cloud Foundation 5.x

Cause

• SDDC Manager is unable to query the NSX AVI Load Balancer version from NSX-T Manager
• NSX-T Manager UI > System > Appliances > NSX Advanced Load Balancer has Version not populated for the NSX AVI nodes
• ALB has been deployed for a WLD in SDDC manager. NSX UI doesn't have support to show ALB details which are deployed for WLDs. It only shows data of ALB that is deployed for management domain. This is the reason you are not seeing details of ALB on NSX UI.

Resolution

1. Reboot the NSX AVI nodes
   
   
2. Navigate to NSX-T Manager UI > System > Appliances > NSX Advanced Load Balancer and check if the Version has populated for the NSX AVI nodes
   
   
   • Note: If NSX Advanced Load Balancer is still missing in the UI then log a case with NSX-T team.
     
     
3. Retry the NSXT upgrade from SDDC Manager

 

If NSX Advanced Load Balancer nodes are not available in NSXT Manager UI then follow the below workaround

 

1. Take snapshot of SDDC Manager VM
2. SSH to SDDC Manager VM with vcf user and su to root
3. Edit /home/vcf/feature.properties file

   vi /home/vcf/feature.properties

4. Set the feature flag to ALB to false

   feature.vcf.nsx.alb=false

   
   If feature do not exist in the file then add the above line
5. Restart lcm service

   systemctl restart lcm

    

NOTE: Enable the flag after NSX and VCenter upgrades are completed, as other ALB APIs also use this feature flag and won't be usable if the flag is off.