Upgrading SDDC to 5.2 when vCenter is running on Version vCenter 8.0u1 build 24005165 with Tanzu
search cancel

Upgrading SDDC to 5.2 when vCenter is running on Version vCenter 8.0u1 build 24005165 with Tanzu

book

Article ID: 377913

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

The main concern here is the TANZU interdependency to the running VCenter build.
VCenter 8.0U1E, build 24005165 supports TANZU Supervisor Cluster versions 1.23, 1.24 and 1.25
Whereas VCenter 8.0 U3A, 8.0.3.00100-24091160 supports only Supervisor Cluster versions 1.26, 1.27 and 1.28.  

In a VVF environment the upgrade can rely on the Autoupgrade function https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-maintenance/GUID-5717618E-65D1-4034-8EC3-F2CB8E494028.html.
The Autoupgrade function allows for 1 version jump/ upgrade to automatically occur, where no middle-ground is available.
Middle-ground = both Vcenter Builds support a common Supervision Cluster version.

Unfortunately, in an SDDC environments, the Autoupgrade function is disabled.
There is no Flag to enable/ disable that function - it's built-in so it can't be bypassed.

Environment

SDDC 5.X environments, where the VCenter was async patched to 8.0U1E, build 24005165.
As we can see the only supported upgrade path for the is listed here https://interopmatrix.vmware.com/Upgrade?productId=2&isHidePatch=false&isHideLegacyReleases=true

VMware VCenter Server 8.0U1e can only be upgraded to VCenter versions 8.0U3b, 8.0U3a and 8.0U3.
The SDDC Upgrade will stop the the VCenter patch, as there is no middle-ground is available 

The upgrade attempt will result in the following message

VCENTER <FQDN> upgrade failed due to errors This Supervisor Cluster is running an unsupported Kubernetes version and becomes incompatible after the vCenter Server upgrade : : { <TASK_ID>:v1.25.6+vmware.1-vsc0.1.5-22042769 } in the bundle precheck.
Remediation Message: Resolve the following errors found in the precheck. Refer to the documentation https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-9FD7FB03-F6F2-4743-AA4E-A4169EC7DEA8.html to identify problems on the host machine that can cause an installation/upgrade to fail.

Log excerpt below
# less -i lcm-debug.log
<Time_and_date> DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=8724e0cf-9934-4c81-aaf0-e87f19e5636e,resourceType=VCENTER,resourceId=<UUID_of-VC>,bundleElementId=50beaa23-0c64-406a-a357-5ffdfb642bbe] [c.v.e.s.l.p.i.vcenter.VCenterCommon,Upgrade-8] vCenterPrecheckStatus for bundle precheck is VC_BUNDLE_LEVEL_PRECHECK_FAILED_WITH_ERRORS
<Time_and_date> [vcf_lcm,0000000000000000,0000,upgradeId=8724e0cf-9934-4c81-aaf0-e87f19e5636e,resourceType=VCENTER,resourceId=<UUID_of-VC>,bundleElementId=50beaa23-0c64-406a-a357-5ffdfb642bbe] [c.v.e.s.l.p.i.vcenter.VCenterCommon,Upgrade-8] errorDescription for bundle precheck is If Supervisor clusters are auto-upgraded it will strand any Tanzu Kubernetes Clusters using Tanzu Kubernetes Release(TKr) versions less than v1.25.0.
Files that cannot be used with Lifecycle Manager  will not be copied from the source. These files include VM guest OS patch baselines, host upgrade baselines and files, and ESXi 6.5 and lower version host patches baselines.
A file-based backup of the vCenter Server was performed 2 hours and 45 minutes ago.
This Supervisor Cluster is running an unsupported Kubernetes version and becomes incompatible after the vCenter Server upgrade :
: { <UUID>:v1.25.6+vmware.1-vsc0.1.5-22042769 }
<Time_and_date>  [vcf_lcm,0000000000000000,0000,upgradeId=8724e0cf-9934-4c81-aaf0-e87f19e5636e,resourceType=VCENTER,resourceId=<UUID_of-VC>,bundleElementId=50beaa23-0c64-406a-a357-5ffdfb642bbe] [c.v.e.s.l.p.i.vcenter.VCenterCommon,Upgrade-8] resolution for bundle precheck is To ensure the Tanzu Kubernetes Clusters are not stranded, Please update all Tanzu Kubernetes Clusters to Tanzu Kubernetes Release (TKr) v1.25.0 or above prior to vCenter upgrade.
Please review VMware Lifecycle Manager Documentation for details
You can proceed with Patching.
VC_BUNDLE_LEVEL_PRECHECK_WCP_UPGRADE_WARNINGS
<Time_and_date>  ERROR [vcf_lcm,0000000000000000,0000,upgradeId=8724e0cf-9934-4c81-aaf0-e87f19e5636e,resourceType=VCENTER,resourceId=<UUID_of-VC>,bundleElementId=50beaa23-0c64-406a-a357-5ffdfb642bbe] [c.v.e.s.l.p.i.vcenter.VCenterCommon,Upgrade-8]
VCENTER <FQDN> upgrade failed due to errors This Supervisor Cluster is running an unsupported Kubernetes version and becomes incompatible after the vCenter Server upgrade :
: { <UUID>:v1.25.6+vmware.1-vsc0.1.5-22042769 } in the bundle precheck.
<Time_and_date> ERROR [vcf_lcm,0000000000000000,0000,upgradeId=8724e0cf-9934-4c81-aaf0-e87f19e5636e,resourceType=VCENTER,resourceId=<UUID_of-VC>,bundleElementId=50beaa23-0c64-406a-a357-5ffdfb642bbe] [c.v.e.s.l.p.i.v.VCenterPatchUpgradeImpl,Upgrade-8] VCenter upgrade failed with VCENTER_BUNDLE_PRECHECK_FAILED_WITH_ERRORS

 

Cause

Resolution

An Out Of Bounds [OOB] upgrade is recommended here, as there are no other options available  
High level overview below.

  1. Follow ELM best practices - All VCenter nodes in ELM to powered off and secured via offline Snapshots 
    https://knowledge.broadcom.com/external/article/313886/vmware-vcenter-in-enhanced-linked-mode-p.html 

  2. Run VCenter Upgrade to BOM target version, to skip additional patching
    https://knowledge.broadcom.com/external/article/314608/correlating-vmware-cloud-foundation-vers.html  - 8.0 U3a 8.0.3.00100-24091160
    Run Upgrade via VAMI [:5480], ISO etc any options that bypasses SDDC.
    https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vcenter-upgrade/GUID-30485437-B107-42EC-A0A8-A03334CFC825.html 

  3. Once that is done, the Autoupgrade function should kick in for the TANZU Supervisor Cluster.
    It will upgrade to 1.26 only, which is expected - only 1 version auto-upgradable, other upgrades need to be manually run.

  4. From SDDC Manager 5.2, inventory sync happens every 24 hours (around 3AM).  
    So post the OOB Upgrade, SDDC Manager inventory will be synced automatically the next day.
    If needed, inventory sync can also be triggered via the below option:
    https://docs.vmware.com/en/VMware-Cloud-Foundation/services/ap-tool/GUID-ED6AEE19-CB7D-44E7-A7D8-D54F8C5CC05D.html?hWord=N4IghgNiBcIJYDsBuBTBAXA9gJwJ4AIBnXBAYxAF8g#inventory-sync-option-9 
Powered by Wolken Software