"Depot connection has issues. Depot Invalid User Credential", SDDC fails to download the updates from online repositories

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• SDDC Manager no longer able to download updates from internet facing VMware by Broadcom public repositories. 
• Depot Settings shows following error:
  
  Some depot connections have issues, which may disrupt software bundle downloads and compatibility data sync

Depot connection has issues. Depot Invalid User Credential
  
  
  
  
  
• Bundle Management shows the Warning "Some depot connection issues may prevent all the latest bundles from being shown and/or downloaded".
  
  
  
  
• SDDC LCM logs shows similar to below snippets :
  
  /var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDTHH:MM:SS.444+0000 DEBUG [vcf_lcm,91ce0966a98f4d1f,1304] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-4] Getting file size for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://<depot_url>/index.v3]
YYYY-MM-DDTHH:MM:SS.444+0000 DEBUG [vcf_lcm,91ce0966a98f4d1f,1304] [c.v.e.s.l.b.d.d.utils.CookieUtils,http-nio-127.0.0.1-7400-exec-4] VCF_DEPOT Depot Http Cookies: []
YYYY-MM-DDTHH:MM:SS.457+0000 DEBUG [vcf_lcm,91ce0966a98f4d1f,1304] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-4] Executing HEAD /<id>/PROD/COMP/SDDC_MANAGER_VCF/index.v3
YYYY-MM-DDTHH:MM:SS.798+0000 DEBUG [vcf_lcm,91ce0966a98f4d1f,1304] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-4] Got response: 401 Unauthorized HTTP/1.1
YYYY-MM-DDTHH:MM:SS.798+0000 ERROR [vcf_lcm,91ce0966a98f4d1f,1304] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-4] Error getting file size, got response: 401 Unauthorized HTTP/1.1
YYYY-MM-DDTHH:MM:SS.799+0000 ERROR [vcf_lcm,91ce0966a98f4d1f,1304] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-4] Got Http error[401] while downloading manifest index [/COMP/SDDC_MANAGER_VCF/index.v3] from <depot>:443 with user

Environment

SDDC Manager 4.5.x
SDDC Manager 5.x

Cause

Public facing repository URLs and authentication mechanisms are changing. Download URLs are no longer common but unique for each customer therefore will require to be re-configured.

Resolution

Follow below steps to update the token in the download URLs :

• Obtain the Download Token from Broadcom Support Portal
• Apply the Token in Download URLs
  • Scripted method
  • Manual method

Download Token

Download the token by following the steps in VCF Authenticated downloads Configuration Update Instructions. If you already have it, apply the same by following below instructions.

Apply the Token in Download URLs

Once the Download Token is ready, same needs to be applied to all the download URLs, any of below approach can be used to configure the token:

Apply the Token - Scripted method

• Authenticated Download Configuration Update Script
• Tools - (Async Patch Tool / Offline Bundle Transfer Utility)

Apply the Token - Manual method

• Take a snapshot of the SDDC Manager VM.
• SSH into the SDDC VM and elevate to root.
• Modify the following properties.
  
  • Edit the application-prod.properties file - vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

set lcm.depot.adapter.host=dl.broadcom.com
set lcm.depot.adapter.remote.rootDir=/<downloadToken>/PROD
set lcm.depot.adapter.remote.repoDir=/COMP/SDDC_MANAGER_VCF
set lcm.depot.adapter.remote.lcmManifestDir=/COMP/SDDC_MANAGER_VCF/lcm/manifest
set lcm.depot.adapter.remote.lcmProductVersionCatalogDir=/COMP/SDDC_MANAGER_VCF/lcm/productVersionCatalog

Note: Replace <downloadToken> with the actual downloaded token id.


• Restart the lcm service for the change to take effect.
  • systemctl restart lcm

Note: This script needs to re-run after every patch upgrades of SDDC Manager.

• Reconnect to the depot from UI (The values entered in the username/password fields will be ignored in the backend, hence it can accept any values).

Additional Information

"Cannot validate proxy configuration. Proxy server didn't allow connection from VCF SDDC Manager to destination URL" or "Can't validate proxy configuration. Destination test URL https://depot.vmware.com returned unexpected HTTP Status. Call VMWARE Support."