VCF Software Depot Download Fails with HTTP 404 / Connection Timeout to Broadcom OAuth Server
search cancel

VCF Software Depot Download Fails with HTTP 404 / Connection Timeout to Broadcom OAuth Server

book

Article ID: 453116

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

During a VMware Cloud Foundation (VCF) Lifecycle Management (LCM) upgrade or bundle download task, bundle downloads fail with 404 Not Found errors in vcf-sddc-lcm. Inspection of the UI reveals the Software Depot component shows a disconnected status under VCF Management. Direct examination of the underlying container logs (vcf-fleet-depot) reveals network connectivity timeouts when reaching out to Broadcom's OAuth authentication endpoint (eapi.broadcom.com).

Environment

 

  • Platform: VMware Cloud Foundation (VCF)  9.x (VMSP / Fleet & SDDC Architecture)

  • Services: VCF LCM (vcf-sddc-lcm), Software Depot Service (vcf-fleet-depot), Fleet Management Services

  • Components Affected: Software Depot Connectivity, Online Bundle Downloads

 

Cause

The vcf-sddc-lcm service fails to fetch update bundles from the local/fleet depot service with a 404 Not Found because the Software Depot service cannot authenticate or retrieve OAuth tokens from Broadcom's external authorization server.

The underlying depot-service container in the vcf-fleet-depot namespace fails to establish an outbound HTTPS connection to [https://eapi.broadcom.com:443](https://eapi.broadcom.com:443), resulting in java.net.ConnectException: Connection timed out. Because the token authentication step fails, depot synchronization cannot occur, leaving the Software Depot service disconnected and unable to locate required bundle binaries.

Log Snippets & Diagnostic Path

1. Analysis of SDDC LCM Logs

Checking LCM logs on the SDDC Manager / Management Instance (/var/log/vmware/vcf/lcm/vcf-lcm.log) indicates a 404 HTTP failure when attempting to query or fetch the target update bundle (e.g., vCenter Appliance 9.1.0.0200.25573614 bundle <BUNDLE_UUID>):
<YYYY-MM-DDThh:mm:ss.fff+0000> DEBUG [vcf_lcm,...] [c.v.e.s.l.b.d.depot.DepotDownloader,Bundle-Downloader-2] Got response: 404 Not Found HTTP/1.1
<YYYY-MM-DDThh:mm:ss.fff+0000> ERROR [vcf_lcm,...] [c.v.e.s.l.b.d.depot.DepotDownloader,Bundle-Downloader-2] Error getting file size, got response: 404 Not Found HTTP/1.1
<YYYY-MM-DDThh:mm:ss.fff+0000> ERROR [vcf_lcm,...] [c.v.e.s.l.s.i.BundleDownloadExceptionHandlerImpl,Bundle-Downloader-2] Bundle download failed
com.vmware.evo.sddc.lcm.model.depot.exception.BundleNotFoundException: Bundle with ID <BUNDLE_UUID> was not found in Software depot https://<FLEET_FQDN>/depot-service/content-gateway and location: /COMP/VCENTER/VMware-vCenter-Server-Appliance-9.1.0.0200.25573614-updaterepo.zip
<YYYY-MM-DDThh:mm:ss.fff+0000> DEBUG [vcf_lcm,...] [c.v.e.s.l.d.c.b.BundleClientImpl,Bundle-Downloader-2] Updating bundle <BUNDLE_UUID> to status FAILED

2. Identifying Containerized Infrastructure (VMSP Context)

Checking the VCF Management UI reveals that services are hosted on containerized control planes (<NODE_PREFIX>-svcs-rt-*). Log in via SSH to the control plane node (<CONTROL_PLANE_IP>) and inspect running Kubernetes pods across all namespaces:

kubectl get pods -A | grep -iE "depot|lcm|fleet"

vcf-fleet-depot   depot-service-<POD_HASH>                                   3/3     Running   0          <AGE>
vcf-fleet-depot   distribution-service-<POD_HASH>                           1/1     Running   0          <AGE>
vcf-fleet-lcm     vcf-fleet-build-service-fleetbuild-<POD_HASH>         1/1     Running   0          <AGE>
vcf-sddc-lcm      vcf-sddc-build-service-sddcbuild-<POD_HASH>     1/1     Running   0          <AGE>

3. Depot Service Logs Analysis

Checking logs directly from the depot-service pod in the vcf-fleet-depot namespace highlights the true underlying connection failure:

kubectl logs -n vcf-fleet-depot depot-service-<POD_HASH> --all-containers -f

output

<YYYY-MM-DDThh:mm:ss.fff+0000> ERROR [download_service,...] [c.v.e.s.l.t.d.impl.TaskServiceImpl,ds-exec-3] Exception happened while processing the binary with id depotConfig
com.vmware.evo.sddc.common.core.error.InternalServerErrorException: Failed to connect to Broadcom OAuth Authorization server.
...
Caused by: com.vmware.evo.sddc.lcm.depot_auth.exceptions.DepotAccessTokenIOException: org.apache.hc.client5.http.ConnectTimeoutException: Connect to https://eapi.broadcom.com:443 failed: Connection timed out
...
Caused by: java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.Net.pollConnect(Native Method)

 

Resolution

  1. Verify Outbound Connectivity: From the control plane node  hosting vcf-fleet-depot, test outbound connection to the Broadcom authentication endpoint:

    curl -I -v https://eapi.broadcom.com

    If this command hangs and times out, network/firewall traffic on port 443 is blocked.

  2. Update Firewall Rules: Ensure network firewalls, proxies, or egress security groups allow outbound HTTPS (Port 443) traffic from all VCF Control Plane and Worker node IPs (<CONTROL_PLANE_IP_RANGE>) to:

    • Host: eapi.broadcom.com

    • Port: 443 (TCP)

  3. Re-validate Depot Connection: Once firewall rules are updated and curl -I -v [https://eapi.broadcom.com](https://eapi.broadcom.com) succeeds:

    • Navigate to the VCF Management UI > Software Depot.

    • Re-enter or test depot credentials to refresh the OAuth token.

    • Verify the Software Depot status changes to Running / Connected.

    • Retry the bundle download or metadata sync from LCM.

 

Additional Information

 

  • Port Requirements: Outbound TCP port 443 connectivity to eapi.broadcom.com and depot.broadcom.com  is mandatory for online software depot downloads and patch availability checks in VCF.

  • Offline Depot Alternative: If enterprise security policies prohibit direct outbound internet access from VCF management nodes, configure an Offline Software Depot using the VMware / Broadcom offline bundle transfer tools.

 

Powered by Wolken Software