VCF Operations Online Depot is not authenticating the Download token - Failed to connect to VMware depot with the provided user credentials. Cause: Internal error while validating credentials

Products

• VMware SDDC Manager
• VCF Operations

Issue/Introduction

• Configuring Online depot with download token from VCF Ops UI -> Fleet Management -> Lifecycle -> VCF Instances -> <VCF Instance Name> -> Depot Settings -> <Enter Download Token> -> Authenticate fails with below error message.
  
  Failed to connect to VMware depot with the provided user credentials. Cause: Internal error while validating credentials
  
  

• LCM logs on SDDC Manager shows similar to below log entries:
  
  /var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDTHH:MM:SS.mmm+0000 INFO  [vcf_lcm, ..] Recoverable I/0 exception (java.net.SocketException) caught when processing request to {s}->https://dl.broadcom.com:443
YYYY-MM-DDTHH:MM:SS.mmm+0000 ERROR [vcf_lcm, ..] Got exception while downloading file [/metadata/productVersionCatalog/v1/productVersionCatalog.json]: Network is unreachable
YYYY-MM-DDTHH:MM:SS.mmm+0000 ERROR [vcf_lcm, ..] Update Depot Settings
com.vmware.evo.sddc.lcm.model.depot.exception.DepotConnectionFailureException: Internal error while validating credentials
  at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotDownloader.validateUser(DepotDownloader.java:566)
    at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotDownloader.validateUser(DepotDownloader.java:594)
    at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotBundleDownloadServiceImpl.validateDepotUserCredential(DepotBundleDownloadServiceImpl.java:757)

Environment

• VCF 9.0

Cause

• This issue is observed when the SDDC Manager appliance is unable to communicate to "dl.broadcom.com" through the proxy due to certificate trust problem.
• Proxy is issuing the certificate from internal CA for the dl.broadcom.com connection and SDDC Manager rejects the connection because the the Root CA or Intermediate CA is not trusted by the SDDC Manager appliance.
• This can be confirmed by checking the following steps :
  
  
  • SSH to SDDC Manager Appliance VM using 'vcf' account
  • Perform curl connection to dl.broadcom.com using below command
    
    curl -v --head https://dl.broadcom.com:443/<Token>/PROD/COMP/SDDC_MANAGER_VCF/index.v3
    
    Note: Replace <Token> variable with the actual download token.
    
    
  • Check the value of cert 'issuer' field under the "Server certificate" section in the curl result. If it is showing Internal CA name, SDDC Manager by default will not trust the certificate.
    
    * Server certificate:
*  subject: CN=dl.broadcom.com
*  start date: <certificate start date>
*  expire date: <certificate expiry date>
*  issuer: <Internal CA Name>

 

Resolution

Add the Internal CA Certificate chain (Root CA and Intermediate CA if any) to the SDDC Manager Appliance keystore by following the steps 1 to 6 in the KB How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores.