vSphere Lifecycle Manager (vLCM) cluster remediation fails with "HTTP Error Code: 403" due to download 7.x VIBs from online repositories.

Products

• VCF Operations
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Attempting to remediate a cluster using vSphere Lifecycle Manager (vLCM), the task fails with "Remediation of cluster failed"
• In the vCenter server navigate to /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log file, you see entries similar to:
  
  YYYY-MM-DDTHH:MM:SS info vmware-vum-server[######] [Originator@6876 sub=EntityImageManager opID=#########-####-######-###########] [EntityImageManager 1370] [SoftwareSpecToStr] image spec(JSON String): {
-->     "add_on": {
-->         "name": "HPE-Custom-AddOn",
-->         "version": "900.0.0.12.2.0-4"
-->     },
-->     "alternative_images": null,
-->     "base_image": {
-->         "version": "9.0.0.0100.24813472"
-->     },
-->     "components": {
-->         "Intel-i40iwn": "1.1.2.6-1vmw.701.0.0.16850804",
-->         "Intel-irdman": "1.4.4.0-1OEM.800.1.0.20143090",
-->         "VMware-VM-Tools": "13.0.5-0.24957456"
-->     },
-->     "hardware_support": null,
-->     "removed_components": null,
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[####] [Originator@6876 sub=RemediateClusterTask] [Task, 424] Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:#########-####-######-########### Task Failed. Error: Error:
 -->  com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.depots.VibDownloadErrorInfo<Failed to download VIB(s): URL: https://dl.broadcom.com/<downloadtoken>/PROD/COMP/ESX_HOST/main/esx/vmw/vib20/i40iwn/VMW_bootbank_i40iwn_1.1.2.6-1vmw.701.0.0.16850804.vib Error: HTTP Error Code: 403>
-->
YYYY-MM-DDTHH:MM:SS warning vmware-vum-server[#######] [Originator@6876 sub=TaskStatsCollector] [taskStatsCollector 190] Task type or creation time not present
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[#######] [Originator@6876 sub=PM.AsyncTask.RemediateClusterTask{20}] [vciTaskBase 1493] SerializeToVimFault fault:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vcIntegrity.lifecycle.depots.VibDownloadErrorInfo",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "URL: https://dl.broadcom.com/<downloadtoken>/PROD/COMP/ESX_HOST/main/esx/vmw/vib20/i40iwn/VMW_bootbank_i40iwn_1.1.2.6-1vmw.701.0.0.16850804.vib Error: HTTP Error Code: 403"
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ],
-->    reason = "vLCM Task failed, see Error Stack for details."
-->    msg = "{
-->     "data": null,
-->     "error_type": "ERROR",

Environment

VMware vCenter Server 8.x

VMware vCenter Server 9.x

VMware vSphere ESXi 8.x

VMware vSphere ESXi 9.x

Cause

This issue occurs if a 7.x component or VIB is manually added to the Image Specification without having the required entitlement.

Resolution

Refer to the steps below to remove 7.x component from the vSphere Lifecycle Manager (vLCM) image spec.

1. Log in to the vSphere Client.

2. Navigate to Hosts and Clusters.

3. Select the affected Cluster and click on the Updates tab.

4. Select Image and click Edit.

5. Locate the section for Components.

6. Find the component causing the error (e.g., Intel-i40iwn version 1.1.2.6-1vmw.701.0.0.16850804).

7. Click the X or Remove icon next to this component.

8. Click Save and Finish to validate the image.

9. Run the Check Compliance task again and proceed with Remediation.

Additional Information

Working with Images at a Cluster or Standalone Host Level