Upgrade Binary downloads in VCF 9.1 fails with HTTP 403 Forbidden Error
search cancel

Upgrade Binary downloads in VCF 9.1 fails with HTTP 403 Forbidden Error

book

Article ID: 441349

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite) VMware SDDC Manager / VCF Installer

Issue/Introduction

  • In a VMware Cloud Foundation (VCF) 9.1 deployment, downloading upgrade binaries (such as NSX and vCenter) from SDDC Manager or VCF Operations fail with an HTTP 403 Forbidden error as shown below



  • Reviewing the task details for the same shows the failure message as per below :



  • The /var/log/vmware/vcf/lcm/lcm.log on the SDDC Manager records explicit HTTP 403 entitlement errors from as observed below: 

    YYYY-MM-DDThh:mm:ss+0000 DEBUG [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.depot.DepotDownloader,Bundle-Downloader-3] Getting file size for [/COMP/<DIRECTORY>/<BUNDLE_NAME>] from URL[https://d1.broadcom.com:443/PROD/COMP/<DIRECTORY/BUNDLE_NAME>]
    YYYY-MM-DDThh:mm:ss DEBUG [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.depot.DepotDownloader,Bundle-Downloader-3] Executing HEAD /PROD/COMP/<DIRECTORY>/<BUNDLE_NAME>
    YYYY-MM-DDThh:mm:ss DEBUG [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.d.DepotDownloader$RetryOnForbidden,Bundle-Downloader-3] Retrying HTTP HEAD /PROD/COMP/<DIRECTORY>/<BUNDLE_NAME> to https://d1.broadcom.com:443
    YYYY-MM-DDThh:mm:ss INFO  [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [o.a.h.c.h.i.c.HttpRequestRetryExec,Bundle-Downloader-3] ex-0000003244 https://d1.broadcom.com:443 responded with status 403; request will be automatically re-executed in 0 NANOSECONDS (exec count 2)
    YYYY-MM-DDThh:mm:ss DEBUG [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.d.DepotDownloader$RetryOnForbidden,Bundle-Downloader-3] Retrying HTTP HEAD /PROD/COMP/<DIRECTORY>/<BUNDLE_NAME> to https://d1.broadcom.com:443
    YYYY-MM-DDThh:mm:ss INFO  [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [o.a.h.c.h.i.c.HttpRequestRetryExec,Bundle-Downloader-3] ex-0000003244 https://d1.broadcom.com:443 responded with status 403; request will be automatically re-executed in 0 NANOSECONDS (exec count 3)
    YYYY-MM-DDThh:mm:ss DEBUG [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.depot.DepotDownloader,Bundle-Downloader-3] Got response: 403 Forbidden HTTP/1.1
    YYYY-MM-DDThh:mm:ss ERROR [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.depot.DepotDownloader,Bundle-Downloader-3] Error getting file size, got response: 403 Forbidden HTTP/1.1
    YYYY-MM-DDThh:mm:ss ERROR [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.d.DepotBundleDownloadServiceImpl,Bundle-Downloader-3] Maximum number of retries without change (3) reached while downloading file: /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles/<BUNDLE_ID>/<BUNDLE_NAME>
    com.vmware.evo.sddc.lcm.bundle.download.depot.exception.HTTPException: 403 Forbidden HTTP/1.1
            at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotDownloader.getFileSizeFromHeadResponse(DepotDownloader.java:473)
            at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotDownloader.lambda$getFileSize$0(DepotDownloader.java:347)
    ...
    YYYY-MM-DDThh:mm:ss ERROR [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.b.d.d.DepotBundleDownloadServiceImpl,Bundle-Downloader-3] Got Http error[403] while downloading bundle [/COMP/<DIRECTORY>/<BUNDLE_NAME>]
    YYYY-MM-DDThh:mm:ss ERROR [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.v.i.b.i.TranslationMessage,Bundle-Downloader-3] can't find resource for bundle java.util.PropertyResourceBundle, key BUNDLE_DOWNLOAD_FAILURE.remedy
    YYYY-MM-DDThh:mm:ss ERROR [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.v.i.b.i.TranslationMessage,Bundle-Downloader-3] can't find resource for bundle java.util.PropertyResourceBundle, key BUNDLE_DOWNLOAD_FAILURE.remedy
    YYYY-MM-DDThh:mm:ss ERROR [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.s.i.BundleDownloadExceptionHandlerImpl,Bundle-Downloader-3] Bundle download failed
    com.vmware.evo.sddc.lcm.model.depot.exception.BundleDownloadFailureException: Download failed for bundle with ID: <BUNDLE_ID>, dl.broadcom.com:443 and location: /COMP/<DIRECTORY>/<BUNDLE_NAME>, http status code: 403
            at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotBundleDownloadServiceImpl.downloadFile(DepotBundleDownloadServiceImpl.java:596)
            at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotBundleDownloadServiceImpl.getDedupBundle(DepotBundleDownloadServiceImpl.java:764)
    ...
    YYYY-MM-DDThh:mm:ss DEBUG [vcf_lcm, 0000000000000000,0000, bundleId=<BUNDLE_ID>] [c.v.e.s.l.d.c.b.BundleClientImpl,Bundle-Downloader-3] Updating bundle <BUNDLE_ID> to status FAILED

  • Manual downloads using the below curl command with the configured token succeed, ruling out general network or firewall blocks, however initiating the download via the VCF Operations or SDDC Manager "Upgrade Binaries" page consistently fails
    curl -o <BUNDLE_NAME> https://dl.broadcom.com:443/<YOUR_DL_TOKEN>/PROD/COMP/<DIRECTORY>/<BUNDLE_NAME>

Environment

VMware Cloud Foundation (VCF) 9.1

Cause

The Broadcom Support Portal download token was generated under the incorrect Site ID and as a result HTTP 403 Forbidden status code was being returned by dl.broadcom.com:443 indicating an authorization rejection.

Resolution

To resolve this issue, the Depot UUID needs to be aligned to the proper Site ID:

  1. Retrieve the Depot UUID from the SDDC Manager > Depot Settings or the VCF Ops > Build > Software Depot.

  2. Log into the Broadcom Support Portal and identify the correct Site ID containing the active VCF 9.1 entitlements with the assistance of Broadcom non-technical support.

  3. Add the Depot UUID to the correct Site ID.

  4. Generate a new download token from the correct Site ID.

  5. Update the token configuration within SDDC Manager or VCF Ops respectively.

Powered by Wolken Software