ESXi Host Compliance or Sync Depot in Lifecycle manager Fails with - A general system error occurred. Failed to download VIB(s): HTTP Error Code: 526

Products

VMware vCenter Server

Issue/Introduction

VVF host upgrades fail during ESXi host compliance:

A general system error occurred. Failed to download VIB(s): URL: http://localhost:1080/depot/PROD/COMP/ESX_HOST/main/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_9.0.2-0.25148076.vib Error: HTTP Error Code: 526;Retry the operation after performing Sync Updates if the VCF Depot is registered to the vCenter.
Navigating to Lifecycle manager > Settings > Download Sources (all URLs display "Not connected").
/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log:

^{yyyy-dd-mmThh:mm:ss ERROR vmware-vum-server [ sub="com.vmware.vcIntegrity.lifecycle.SyncDepotsTask"] [SyncDepotsTask] Failed to sync depots. Merged depot content is invalid: Error:}
^{yyyy-dd-mmThh:mm:ss ERROR vmware-vum-server [sub="com.vmware.vcIntegrity.lifecycle.SyncDepotsTask"] [Task, 478] Task:com.vmware.vcIntegrity.lifecycle.SyncDepotsTask ID:520a0ed8-c97e-1a28-1a25-a9065c0a4d00. Task Failed. Error: Error:}
^{--> com.vmware.vapi.std.errors.error}
^{--> Messages:}
^{--> com.vmware.vcIntegrity.lifecycle.depotContent.ValidationErrorInfo<A depot is inaccessible. URL: 'http://localhost:1080/depot/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml'. Error: 'HTTP Error Code: 526'. Make sure an official depot source is used and verify connection to the depot.>}
^-->
^{yyyy-dd-mmThh:mm:ss.552Z INFO vmware-vum-server [sub="PM.AsyncTask.SyncDepotsTask{74}"] [vciTaskBase 1493] SerializeToVimFault fault:}
^{--> (vmodl.fault.SystemError) {}
^{--> faultCause = (vmodl.MethodFault) null,}
^{--> faultMessage = (vmodl.LocalizableMessage) [}
^{--> (vmodl.LocalizableMessage) {}
^{--> key = "com.vmware.vcIntegrity.lifecycle.depotContent.ValidationErrorInfo",}
^{--> arg = (vmodl.KeyAnyValue) [}
^{--> (vmodl.KeyAnyValue) {}
^{--> key = "1",}
^{--> value = "http://localhost:1080/depot/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml"}
^{--> },}
^{--> (vmodl.KeyAnyValue) {}
^{--> key = "2",}
^{--> value = "HTTP Error Code: 526"}
^{--> }}
^{--> ],}
^{--> message = <unset>}
^{--> }}
^{--> ],}
^{--> reason = "vLCM Task failed, see Error Stack for details."}
^{--> msg = "{}
^{--> "data": null,}
^{--> "error_type": "ERROR",}
^{--> "messages": [}
^{--> {}
^{--> "args": [}
^{--> "http://localhost:1080/depot/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml",}
^{--> "HTTP Error Code: 526"}
^{--> ],}
^{--> "default_message": "A depot is inaccessible. URL: 'http://localhost:1080/depot/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml'. Error: 'HTTP Error Code: 526'. Make sure an official depot source is used and verify connection to the depot.",}
^{--> "id": "com.vmware.vcIntegrity.lifecycle.depotContent.ValidationErrorInfo"}
^{--> }}

A curl test to the depot URL fails, returning an Invalid SSL Certificate error:

^{# curl -v http://localhost:1080/depot/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml}
^{* Uses proxy env variable no_proxy ==}
^{* Host localhost:1080 was resolved.}
^{* IPv6: ::1}
^{* IPv4: 127.0.0.1}
^{* Trying [::1]:1080...}
^{* Established connection to localhost (::1 port 1080) from ::1 port 53496}
^{* using HTTP/1.x}
^{GET /depot/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml HTTP/1.1}^{HTTP/1.1 526 Invalid SSL Certificate}

Environment

VMware vSphere Foundation (VVF) 9.x
VMware vCenter Server 9.x
VMware vSphere ESXi 9.x

Cause

The VCF Fleet Depot Service (FDS) certificate is incorrect on the VMware vCenter Server. The FDS is registered on the vCenter Server, but the TLS is using the local VMCA root certificate instead of the actual issuing CA from the VCF Operations node.
As a result, we receive HTTP Error Code 526 when downloading VIBs.

Resolution

Following APIs can be used for updating the FDS certificates in the vCenter - https://developer.broadcom.com/xapis/vsphere-automation-api/9.1/vcenter-lcm/vcenter-lcm-depot-services/

1. Obtain an API session ID on the vCenter Server by running the following command (replace the placeholder with the actual administrator password):

SESSION_ID=$(curl --request POST --url https://localhost/api/session -u '[email protected]:<password>' -k | tr -d '"')

2. Backup the current FDS configuration to a JSON file:

curl --request GET --url https://localhost/api/vcenter/lcm/depot/services --header "vmware-api-session-id: $SESSION_ID" -k | jq > /tmp/depot-services-backup.json

3. Edit the /tmp/depot-services-backup.json file and replace the certificates field with the new certificate chain presented by the FDS.

4. Update the FDS configuration using the modified JSON file (a 204 HTTP response code indicates a successful update):

curl --request PUT --url https://localhost/api/vcenter/lcm/depot/services --header 'Content-Type: application/json' --header "vmware-api-session-id: $SESSION_ID" -d @/tmp/depot-services-backup.json -k -w "\nHTTP %{http_code}\n"