Cluster remediation is getting stuck at 2%

Products

VMware vSphere ESXi VMware vCenter Server VMware SDDC Manager / VCF Installer

Issue/Introduction

vCenter Lifecycle Manager (vLCM) is being used in single image mode. After upgrading vCenter and uploading the image to the latest version, an error message such as below will present itself in the GUI:

"Failed to validate the checksum of VIB '/tmp/offlineBundle6lcz9d/vib20/example/example_8.0.3-#####.vib', because the computed checksum differs from the expected one stored in metadata. Make sure the specified VIB is NOT corrupted and the expected checksum stored in metadata is correct."

Cluster remediation is getting stuck at 2% with error " A general system error occurred: Cannot download VIB "
When viewing for updated Vendor-addon's or updated components, the versions are dated because there is no internet access for vLCM to download recent copies of packages.
In the SDDC environment when initiating the ESXI hosts remediation we see below errors:

"Failed to validate the checksum of VIB '/tmp/offlineBundleAVL2Hj/vib20/example/example.vib', because the computed checksum differs from the expected one stored in metadata. Make sure the specified VIB is NOT corrupted and the expected checksum stored in metadata is correct"

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

VCF 5.x

Cause

This is due to corrupt metadata in the VUM database backing vCLM.

Resolution

The following operational workarounds are available to mitigate this issue :

1. Download and upload newer copies of the VIB in the error message and select the updated version for components by following the below steps :

Import the Update: Download the updated component offline bundle (.zip) and import it into the local vSphere Lifecycle Manager (vLCM) repository via Menu > Lifecycle Manager > Actions > Import Updates.
Modify the Cluster Image: Navigate to the target cluster's Updates > Image tab, click Edit, select or add the newly imported component version, and click Save.

2. Download and upload a newer copy of the vendor-addon package which should include all applicable VIB by following the below steps :

Import the Vendor Add-on: Download the latest vendor add-on offline bundle (.zip) from your hardware vendor (e.g., Dell, HPE) and import it into the vSphere Lifecycle Manager (vLCM) repository via Menu > Lifecycle Manager > Actions > Import Updates.
Modify the Cluster Image: Navigate to the target cluster's Updates > Image tab and click Edit. Locate the Vendor Addon section, select the newly imported version from the dropdown menu, and click Save.

In certain cases, the metadata that has been corrupted will be on a VIB that is already at the latest version (Example: Dell FIST VIB). In this case, a reset of the VUM database is required

1. Use the following KB to reset the VUM database: Resetting VMware Update Manager Database in vCenter Server Appliance 6.5/6.7/7.0/8.0
2. Re-upload the latest image, and vendor addon to vLCM
3. Scan and remediate the host