"Failed to extract image from the host: no stored copy available for inactive VIB" when extracting an image from ESX 9.0 that was upgraded to 9.0 using the ISO upgrade method

Products

• VMware vSphere ESXi

Issue/Introduction

After performing a successful upgrade of an ESXi host using an ISO image, subsequent operations that rely on host seeding, such as a host image extraction or deployment of vCenter in a management domain, may fail. The process fails because VIBs that were reserved from the previous installation are deleted during the ISO upgrade process, leading to an inconsistent host image profile.

var/log/lifecycle.log (ESX)

Er(11) lifecycle[2122910] HostSeeding:1136 Extract depot failed: ('VMW_bootbank_irdman_1.4.0.3-1vmw.900.0.24755229', 'Failed to add reserved VIB VMW_bootbank_irdman_1.4.0.3-1vmw.900.0.24755229: not found in the reserved VIB cache storage')
Er(11) lifecycle[2122910] imagemanagerctl:399 Extract depot failed.
Er(11) lifecycle[2122910] imagemanagerctl:122 [ReservedVibExtractError]
Er(11) lifecycle[2122910] imagemanagerctl:127 esximage.ImageManager.HostSeeding.VibNotInCacheError: VIB VMW_bootbank_irdman_1.4.0.3-1vmw.900.0.24755229 is not available in cached locations

/var/log/vmware/vcf/domainmanager/ci-installer-##/workflow_###/vcsa-cli-installer.log (VCF Installer)

message = "An internal error occurred: 'Error:\n   com.vmware.vapi.std.errors.error\nMessages:\n   com.vmware.vcIntegrity.lifecycle.ExtractDepotTask.HostExtractDepotFailed<Extraction of image from host ###### failed.>\n   com.vmware.vcIntegrity.lifecycle.EsxImage.ReservedVibExtractError<Failed to extract image from the host: no stored copy available for inactive VIB <vib name>\n'"

Environment

VCF 9.x

Cause

The issue is triggered by a specific sequence of events during an ISO-based upgrade.

1. Prior to upgrading, higher-version VIBs are installed onto an ESXi host, replacing the ones that came with the base image. This action causes the original, lower-version VIBs to be marked as "reserved" and cached in the /var/vmware/lifecycle/hostSeed/reservedVibs/ directory.
2. The ESX host is upgraded using an ISO image. The target ESXi version in the ISO still contains the older versions of the VIBs.
3. During the ISO upgrade, the installer logic deletes the entire reservedVibs directory, permanently losing the VIBs that were reserved in Step 1.
4. The installer then copies the reserved VIBs from the target ISO's resvib.tgz archive if one exists. This archive may contain a different set of reserved vibs than the ones that got deleted. 
5. After the upgrade, the host's image profile is inconsistent. When a process like host seeding is initiated, it fails because it cannot find the expected reserved VIB files.

Resolution

A permanent fix is planned for a future patch release. In the interim, two workarounds are available.

Workaround 1: Re-Image the Host
This method involves reinstalling the ESXi host and is more disruptive.

1. Reinstall the host directly using the target custom ISO image (e.g., the HPE custom image).
2. After the installation and first reboot, manually install the higher-version components that are required for your environment. This will require a second reboot.
3. Proceed with the host image extractions, Management domain deployment, or other host seeding operations.

 

Workaround 2: Recover a Host in the Failed State 
This method recovers the host without requiring a full reinstallation.

1. Determine the component(s) that were causing issues from the log entries in lifecycle.log. 
2. Obtain the .vib files for the components that were originally in the base image. These files can be found in the offline depot .zip file for the original ESXi version (e.g., the vanilla 9.0 GA offline depot).
3. Copy the required .vib files to the reserved VIBs directory on the affected ESXi host: /var/vmware/lifecycle/hostSeed/reservedVibs/
4. Retry the failed operation. No reboot of the ESXi host is required.
   
   Note: Determining the problematic components may require multiple host seeding tries as the operation stops at the first failure. Users need to repeatedly run all 4 steps until it succeeds.