Failed to add new token-based URL in Lifecycle Manager, error "is invalid or cannot be reached now"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• When adding new token-based URL on vCenter Lifecycle Manager it failed with following error.

The download source https://dl.broadcom.com/<Token ID>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml is invalid or cannot be reached now.

• To verify certificate introspection, follow these steps: 

vCenter should be allowed to reach new URL http://dl.broadcom.com/ on proxy/firewall, to validate if vCenter can access this URL run following cmd from vCenter.

curl -vv https://dl.broadcom.com

Expected output,

* Host dl.broadcom.com:443 was resolved.

* IPv6: 2a06:98c1:58::a5, 2606:4700:7::a5

* IPv4: 162.159.140.167, 172.66.0.165

*  Trying 162.159.140.167:443...

* Connected to dl.broadcom.com (162.159.140.167) port 443

 

• To verify whether the certificate is being replaced with one signed by an untrusted Certificate Authority. 

 curl -vv https://dl.broadcom.com/<Token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml

* Host dl.broadcom.com:443 was resolved.

* IPv6: 2a06:98c1:58::a5, 2606:4700:7::a5

* IPv4: 162.159.140.167, 172.66.0.165

*   Trying 162.159.140.167:443...

* Connected to dl.broadcom.com (162.159.140.167) port 443

* ALPN: curl offers http/1.1

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

*  CAfile: /etc/pki/tls/certs/ca-bundle.crt

*  CApath: none

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (OUT), TLS alert, unknown CA (560):

* SSL certificate problem: self-signed certificate in certificate chain

* Closing connection

curl: (60) SSL certificate problem: self-signed certificate in certificate chain

More details here: https://curl.se/docs/sslcerts.html 

• • You may also see the following entries in /var/log/vmware/vmware-updatemgr/vmware/vmware-vum-server-###.log:
    
    YYYY-MM-DDTHH:MM:SS.###-##:## verbose vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] * Connected to dl.broadcom.com (###.###.###.###) port 443 (####)
YYYY-MM-DDTHH:MM:SS.###-##:## verbose vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] * ALPN: offers http/1.1
YYYY-MM-DDTHH:MM:SS.###-##:## verbose vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
YYYY-MM-DDTHH:MM:SS.###-##:## verbose vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] *  CAfile: /etc/pki/tls/certs/ca-bundle.crt
YYYY-MM-DDTHH:MM:SS.###-##:## verbose vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] *  CApath: /etc/ssl/certs
YYYY-MM-DDTHH:MM:SS.###-##:## verbose vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] * SSL certificate problem: self signed certificate in certificate chain
YYYY-MM-DDTHH:MM:SS.###-##:## verbose vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] * Closing connection ###
YYYY-MM-DDTHH:MM:SS.###-##:## error vmware-vum-server[#####] [Originator@#### sub=httpDownload] [httpDownloadPosix ###] curl_easy_perform() failed: cURL Error: SSL peer certificate or SSH remote key was not OK,
SSL certificate problem: self signed certificate in certificate chain
    
    

 

• This command displays certificate issuer details. The expected output should indicate that dl.broadcom.com is signed by a public Certificate Authority (Global CA).

echo | openssl s_client -connect dl.broadcom.com:443 2>/dev/null -showcerts | sed -n '/^-----BEGIN CERTIFICATE-----/,/^-----END CERTIFICATE-----/p' | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout

Expected output,
subject=CN=0b227458.sni.cloudflaressl.com
issuer=C=US, O=Google Trust Services, CN=WE1

subject=C=US, O=Google Trust Services, CN=WE1
issuer=C=US, O=Google Trust Services LLC, CN=GTS Root R4

subject=C=US, O=Google Trust Services LLC, CN=GTS Root R4
issuer=C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA

If the issuer in the output differs, it indicates that the certificate has been intercepted.

Environment

VMware vCenter Server 8.0x
VMware vCenter Server 7.0.x

Cause

Issue can be cause by one of the reason:

• vCenter server is unable to reach Broadcom URL
• The SSL connection between vCenter and Broadcom, and certificate is being decrypted and re-encrypted by the SSL introspection software.

Resolution

To prevent this issue, ensure that communication between the vCenter Server and https://dl.broadcom.com is not decrypted and re-encrypted by the SSL introspection software, but can instead pass untouched.