Offline Depot configuration on SDDC Manager fails with "Failed to connect to VMware depot"
search cancel

Offline Depot configuration on SDDC Manager fails with "Failed to connect to VMware depot"

book

Article ID: 427180

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

  • Configuring the Offline Depot in the SDDC Manager UI fails with the following error: Failed to connect to VMware Depot with the provided user credentials. Cause: {0}

  • From the var/log/vmware/vcf/lcm/lcm-debug.log reports 404 Not Found and fails to find valid certificates.


##-##-##T##:##:##.##+0000 DEBUG [vcf_lcm,e67d5c7a68224f24,7316] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-8] Getting file size for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://xxx.org:443/DownloadToken/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
##-##-##T##:##:##.##+0000 DEBUG [vcf_lcm,e67d5c7a68224f24,7316] [c.v.e.s.l.b.d.d.utils.CookieUtils,http-nio-127.0.0.1-7400-exec-8] VCF_DEPOT Depot Http Cookies: []
##-##-##T##:##:##.##+0000 DEBUG [vcf_lcm,e67d5c7a68224f24,7316] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-8] Executing HEAD /DownloadToken/PROD/COMP/SDDC_MANAGER_VCF/index.v3
##-##-##T##:##:##.##+0000 DEBUG [vcf_lcm,e67d5c7a68224f24,7316] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-8] Got response: 404 Not Found HTTP/1.1
##-##-##T##:##:##.##+0000 ERROR [vcf_lcm,e67d5c7a68224f24,7316] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-8] Error getting file size, got response: 404 Not Found HTTP/1.1
##-##-##T##:##:##.##+0000 ERROR [vcf_lcm,e67d5c7a68224f24,7316] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-8] Got Http error[404] while downloading manifest index [/COMP/SDDC_MANAGER_VCF/index.v3] from xxx.org:443 with user vmware
##-##-##T##:##:##.##+0000 ERROR [vcf_lcm,e67d5c7a68224f24,7316] [c.v.v.l.r.a.c.v.s.DepotSettingsController,http-nio-127.0.0.1-7400-exec-8] Update Depot Settings
com.vmware.evo.sddc.lcm.model.depot.exception.DepotConnectionFailureException: Internal error while validating credentials

.

##-##-##T##:##:##.##+0000 WARN  [vcf_lcm,695f8be6ae587829480be1fc20ac6772,f9ac] [c.v.e.s.l.c.s.BundleManifestDownloadScheduler,Scheduled-10] Failed to download index file and all bundle manifest files from it. Error:
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)

  • Direct connection test from SDDC Manager to the Offline Depot fails with a 404 Not Found error:

curl -I -k -u <depot_username>:<depot_user_password> https://<Offline_Depot_FQDN>/COMP/SDDC_MANAGER_VCF/index.v3
HTTP/1.1 404 Not Found
Server: Apache/2.4.62 (Red Hat Enterprise Linux)

Environment

VMware Cloud Foundation 5.2.x

Cause

This issue occurs due to three main reasons:

  • Missing Trust: The Offline Depot web server certificate has not been added to the SDDC Manager trust store.
  • Incorrect Pathing: The parameters in /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties do not include the required /offline_depot/ prefix, preventing the web server from mapping the URL to the physical directory.
    • less /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

################### LCM DEPOT PROPERTIES ########################
lcm.depot.adapter.host=dl.broadcom.com
lcm.depot.adapter.port=443
lcm.depot.adapter.remote.rootDir=/DownloadToken/PROD <---- Missing /offline_depot/
lcm.depot.adapter.remote.repoDir=/COMP/SDDC_MANAGER_VCF
lcm.depot.adapter.local.baseDir=/nfs/vmware/vcf/nfs-mount/bundle/depot/local
lcm.depot.adapter.enableBundleSignatureValidation=true
lcm.depot.adapter.certificateCheckEnabled=true
lcm.depot.adapter.remote.index.filename=index.v3
lcm.depot.adapter.softwareCompatibilitySetsFile=softwareCompatibilitySets.json
lcm.depot.adapter.partnerBundleMetadata.updated.filename=vxrailPartnerBundleMetadata.json
lcm.depot.credential.file.path=*******
lcm.depot.bundleElement.patchFile.checksumValidation=true
lcm.depot.adapter.lcmManifestFile=lcmManifest.json
lcm.depot.adapter.remote.lcmManifestDir=/COMP/SDDC_MANAGER_VCF/lcm/manifest

  • Missing Web Server Alias: Without a defined Alias in the Apache/Web Server configuration, the server cannot map the URL path to the actual physical directory on the disk, resulting in a 404 Not Found.

 

Resolution

To resolve the issue follow through the below steps.

  1. Fix Web Server 404 and 403 Errors
    • Fix 404 Not Found: Ensure an Alias is configured on your Web Server (e.g., Apache) to expose the directory where binaries are stored.
    • Fix 403 Forbidden: If a 403 error appears, ensure the web server user has the necessary read and execute permissions on the offline depot folders.

  2. Update LCM Depot Properties
    1. SSH to SDDC Manager as root.
    2. Edit /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties.
    3. Ensure the rootDir reflects the /offline_depot/ path:

      ############ LCM DEPOT PROPERTIES ############
      lcm.depot.adapter.host=dl.broadcom.com
      lcm.depot.adapter.remote.rootDir=/offline_depot/<DownloadToken>/PROD
      lcm.depot.adapter.remote.repoDir=/COMP/SDDC_MANAGER_VCF

  3. Import Certificates to SDDC Manager & Java Trust Store
    1. Retrieve Certificate:
      • Run this command from SDDC Manager to get the SSL certificate:
        • openssl s_client -connect "offlinedepot IP/FQDN":443
    2. Take a snapshot of the SDDC Manager VM.
    3. Copy File: Use a file transfer utility to copy the certificate to /tmp on SDDC Manager.
    4. Access Shell: SSH as vcf user, then run su - to become root.
    5. Obtain Key:
      • KEY=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
      • echo $KEY
        (Sample output: iDxxxxxxxxxxxxxxx6_m)
    6. Import to SDDC Trust Store:
      • keytool -importcert -alias ProxyServer -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass $KEY (Type yes when prompted)
    7. Import to Java Trust Store:
      • keytool -importcert -alias ProxyServer -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
  4. Restart and Verify
    1. Restart Services:
      • /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
    2. Verify Trust Store:
      • keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY
    3. Retry to configure the Offline Depot.

 

Powered by Wolken Software