SDDC Manager Unable to Connect to Depot

Products

VMware SDDC Manager VMware Cloud Foundation 5.x VMware Cloud Foundation

Issue/Introduction

The following error is observed while connecting the SDDC Manager to the Depot: "Depot connection has issues. Depot Unknown Error."

After executing the steps in either of below KBs, VMware depot is not getting connected and is failing with below error:

Failed to connect to VMware depot with the provided user credentials. Cause: {0}.

Authenticated Download Configuration Update Script Depot connection has issues. Depot Invalid User Credential", SDDC fails to download the updates from online repositories
Depot connection has issues. Depot Invalid User Credential", SDDC fails to download the updates from online repositories

In/var/log/vmware/vcf/lcm/lcm-debug.log appears "Internal error while validating credentials" during Depot connection in SDDC Manager:

INFO [vcf_lcm,92############78,0da5] [c.v.v.l.r.a.c.v.s.DepotSettingsController,http-nio-127.0.0.1-7400-exec-1] Update Depot Settings: { "vmwareAccount": { "username": "<depot_user>", "password": "*****" } }
INFO [vcf_lcm,92############78,0da5] [c.v.e.s.l.s.i.DepotSettingsServiceImpl,http-nio-127.0.0.1-7400-exec-1] Updating VCF_DEPOT account
DEBUG [vcf_lcm,92############78,0da5] [c.v.e.s.l.b.d.d.utils.CookieUtils,http-nio-127.0.0.1-7400-exec-1] VCF_DEPOT Depot Http Cookies: []
DEBUG [vcf_lcm,92############78,0da5] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-1] Getting file size for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://dl.broadcom.com:443/<TOKEN_GENERATED_FROM_SUPPORT_PORTAL>/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
DEBUG [vcf_lcm,92############78,0da5] [c.v.e.s.l.b.d.d.utils.CookieUtils,http-nio-127.0.0.1-7400-exec-1] VCF_DEPOT Depot Http Cookies: []
DEBUG [vcf_lcm,92############78,0da5] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-1] Executing HEAD /<TOKEN_GENERATED_FROM_SUPPORT_PORTAL>/PROD/COMP/SDDC_MANAGER_VCF/index.v3
INFO [vcf_lcm,92############78,0da5] [o.a.h.c.h.i.c.HttpRequestRetryExec,http-nio-127.0.0.1-7400-exec-1] Recoverable I/O exception (java.net.SocketException) caught when processing request to {s}->https://dl.broadcom.com:443
ERROR [vcf_lcm,92############78,0da5] [c.v.e.s.l.b.d.depot.DepotDownloader,http-nio-127.0.0.1-7400-exec-1] Got exception while downloading manifest index [/COMP/SDDC_MANAGER_VCF/index.v3]: Network is unreachable
ERROR [vcf_lcm,92############78,0da5] [c.v.v.l.r.a.c.v.s.DepotSettingsController,http-nio-127.0.0.1-7400-exec-1] Update Depot Settings
com.vmware.evo.sddc.lcm.model.depot.exception.DepotConnectionFailureException: Internal error while validating credentials
at com.vmware.evo.sddc.lcm.bundle.download.depot.DepotDownloader.validateUser(DepotDownloader.java:410)

Errors in Log /var/log/vmware/vcf/lcm/lcm.log appears as "Network is unreachable."

YYYY-MM-DD0T03:34:02 INFO [vcf_lcm,<ID_REDACTED>] [DepotSettingsController] Fetching depot settings
YYYY-MM-DD0T03:34:02 INFO [vcf_lcm,<ID_REDACTED>] [BundleDownloadController] Get the most recent bundle download
YYYY-MM-DD0T03:34:21 WARN [vcf_lcm,<ID_REDACTED>] [ProxyAwareHttpExecutor] Failed 3 of 3 attempts to send payload from VMware server
Reason: ConnectionException: Network is unreachable
YYYY-MM-DD0T03:34:32 ERROR [vcf_lcm,<ID_REDACTED>] [BundleManifestDownloadScheduler] Exception while polling for bundle manifests java.net.SocketException: Network is unreachable

While checking curl connectivity with "dl.broadcom.com" command output shows "Network is unreachable"

curl -v --head https://dl.broadcom.com:443/<Token>/PROD/COMP/SDDC_MANAGER_VCF/index.v3

* Host dl.broadcom.com:443 was resolved.
* IPv6: XXX
* IPv4: XXX
* Immediate connect fail for XXX: Network is unreachable
* Trying XXX :443...
* ALPN: curl offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, bad certificate (554):
* SSL certificate problem: certificate is not yet valid
* closing connection #0 curl: (60) SSL certificate problem: certificate is not yet valid More details here: https://curl.se/docs/sslcerts.html

Environment

VMware SDDC Manager

Cause

This issue is observed when the SDDC Manager appliance is unable to communicate to "dl.broadcom.com"
The new download URL "dl.broadcom.com" is not allowed in the firewall.

Resolution

To resolve this issue allow https://dl.broadcom.com:443/<TOKEN_GENERATED_FROM_SUPPORT_PORTAL> in the firewall

Workaround:

Import Firewall's proxy server certificate in SDDC Manager trust store - Refer How to import Proxy server certificate to SDDC manager trust store.

You may also add the CA certificates of the dl.broadcom.com into the trusted roots store of the SDDC Manager via API explorer.

Additional Information

The same error message can be shown if the old public repositories are still used in the VCF environment. To update the depot settings with unique Token based URL follow the steps from KB: VCF authenticated downloads configuration update instructions