Host seeding fails after upgrading from an OEM ESXi image to a higher version of OEM ESXi image

Products

• VMware vSphere ESXi

Issue/Introduction

When a host running an OEM image of version, say 7.0.2, is upgraded to a higher version of OEM image, say 7.0.3, the host cannot be used as a seed to create a new vLCM cluster. The offline depot creation on the host fails with a traceback similar to the one shown below. Host seeding was a feature introduced in 7.0U2. This KB applies to any two ESXi versions X and Y where X >= 7.0U2 and Y > X

[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: HostSeeding:956 ERROR Extract depot failed: ('VMW_bootbank_bnxtroce_216.0.58.0-23vmw.703.0.0.18644231', 'Failed to add reserved VIB VMW_bootbank_bnxtroce_216.0.58.0-23vmw.703.0.0.18644231:not found in the reserved VIB cache storage')
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:373 ERROR Extract depot failed.
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:152 ERROR [ReservedVibExtractError] ('VMW_bootbank_bnxtroce_216.0.58.0-23vmw.703.0.0.18644231', 'Failed to add reserved VIB VMW_bootbank_bnxtroce_216.0.58.0-23vmw.703.0.0.18644231:not found in the reserved VIB cache storage')
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR Traceback (most recent call last):
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/Depot.py", line 821, in GenerateVib
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166:imagemanagerctl:157 ERROR     resVibPath = resVibCache.getVibLocation(vibobj.id)
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 1059, in getVibLocation
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR     raise VibNotInCacheError('VIB %s is not available in cached locations'
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR vmware.esximage.ImageManager.HostSeeding.VibNotInCacheError: VIB VMW_bootbank_bnxtroce_216.0.58.0-23vmw.703.0.0.18644231 is not available in cached locations
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR During handling of the above exception, another exception occurred:
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR Traceback (most recent call last):
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR File "/usr/lib/vmware/lifecycle/bin/imagemanagerctl.py", line 371, in depots
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR     HostSeeding.InstalledImageInfo().extractDepot(task)
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 942, in extractDepot
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR     Depot.DepotFromImageProfile(newProfile, depotDir,
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/Depot.py", line 1026, in DepotFromImageProfile
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR     vibdownloadfn(localfn, allRelatedVibs[vibid],
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/Depot.py", line 824, in GenerateVib
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR     raise Errors.ReservedVibExtractError(vibobj.id,
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2101166: imagemanagerctl:157 ERROR vmware.esximage.Errors.ReservedVibExtractError: ('VMW_bootbank_bnxtroce_216.0.58.0-23vmw.703.0.0.18644231', 'Failed to add reserved VIB VMW_bootbank_bnxtroce_216.0.58.0-23vmw.703.0.0.18644231: not found in the reserved VIB cache storage') storage')
 

 

Environment

VMware vSphere ESXi 7.0.2

Cause

During installation of ESXi, the reserved vibs are preserved on the host. However, while upgrading, if there is any reserved vib in the running image that matches a reserved vib in the upgrade target image, such reserved vibs previously preserved are removed due to a bug in the business logic.

Terminology: Reserved vibs are the vibs that is present in the base (stock) image provided by VMware but are overridden by higher version async vibs packaged by OEM.
 

Resolution

The issue has been resolved in 8.0GA.

Workaround:

Interactive or scripted upgrade can be used as a workaround. Below are the links.
Interactive upgrade 
Scripted upgrade 

Users running 7.x vSphere releases can use interactive or scripted upgrade as a workaround.