Cannot upgrade ESXi from SDDC manager due to stale ESXi custom image configuration - VCF 5.x

Products

• VMware SDDC Manager

Issue/Introduction

Upgrade workflow on SDDC manager fails with error "Incompatible patch or upgrade files. Please verify that the patch file is compatible for host".

ESXi VUA log

-->       <name>MISSING_DEPENDENCY_VIBS</name>
-->       <expected>
-->       </expected>
-->       <found>
-->         <value>HPE_bootbank_sut_701.2.9.0.44-1OEM.701.0.0.16850804</value>
...
-->         <value>HPE_bootbank_fc-enablement_700.3.8.0.6-1OEM.700.1.0.15843807</value>
-->         <value>HPE_bootbank_amsd_701.11.8.0.15-1OEM.701.0.0.16850804</value>
-->       </found>
-->       <result>ERROR</result>

 

SDDC LCM log

[YYYY-MM-DDTHH:MM:SS] INFO  [vcf_lcm,0000000000000000,0000,upgradeId=######-######-######-######,resourceType=ESX_HOST,resourceId=######-######-######-######,bundleElementId=4######-######-######-######] [c.v.e.s.l.p.i.e.EsxVumPrimitiveImpl,Upgrade-3] ESX custom image upgrade spec is { "esxCustomImageSpecList": [ { "bundleId": "######-######-######-######", "targetEsxVersion": "7.0.3-19193900", "useVcfBundle": false, "customIsoAbsolutePath": "/nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries/VMware-ESXi-7.0.3-19193900-HPE-703.0.0.10.8.1.3-Jan2022.iso" } ] }

Environment

VMware SDDC Manager 5.x
VMware vSphere ESXi 7.x/8.x

Cause

This issue is due to the ESXi custom image configuration used during the VCF deployment, which installs some 3rd-party VIBs . When upgrading the ESXi, it uses the standard ISO, which doesn't include these 3rd part VIBs

Resolution

There are 2 methods

Snapshot SDDC Manager VM before making changes

Method #1: Follow the document to upgrade ESXi with custom ISO
(1) Download OEM ISO from support.broadcom.com
(2) Upload ISO to SDDC manager
(3) Create a custom ESXi ISO configuration file. You may need KB for reference.
(4) Modify /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties 
(5) Upgrade the ESXi

More details can be found at here.

 

Method #2: Follow the KB to remove 3rd part VIBs manually.

Note: You may need to install them after the upgrade is completed.