Lifecycle Manager fails with "The download source dl.broadcom.com is invalid or cannot be reached now." due to ssl interception

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
Adding url to download patch definitions in the vSphere Web Client fails with the following symptoms:

vSphere Web Client (HTML5):

• When adding url to download patches in Home > Update Manager > [vum-server]  > Manage > Download Settings in vCenter Server 6.5/6.7 or in Home > Lifecycle Manager > Settings > Patch Setup in vCenter Server 7.x/8.x
• After clicking the New button, entering the url with token (https://dl.broadcom.com/<downloadToken>/PROD/COMP/VCENTER/...) it fails with a message when clicking on save with:

  The download source https://dl.broadcom.com/<Download Token>/PROD/COMP/VCENTER/... is invalid or cannot be reached now.

vCenter (BASH) shell:

• Attempting to connect with curl results in an error:

  # curl https://dl.broadcom.com/<Download Token>//PROD/COMP/VCENTER/...
  
  curl: (60) SSL certificate problem: unable to get local issuer certificate
  More details here: https://xxxx.xxxx.xx/docs/sslcerts.html
  
  curl failed to verify the legitimacy of the server and therefore could not
  establish a secure connection to it. To learn more about this situation and
  how to fix it, please visit the web page mentioned above.

In /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log you find the following entries:

[2019-09-20 21:00:11:185 'httpDownload' 140052601693952 ERROR]  [httpDownloadPosix, 649] curl_easy_perform() failed: cURL Error: Peer certificate cannot be authenticated with given CA certificates, SSL certificate problem: unable to get local issuer certificate
[2019-09-20 21:00:11:185 'DownloadMgr' 140052601693952 ERROR]  [downloadMgr, 629] Executing download job {140052105550256} throws error: curl_easy_perform() failed: cURL Error: Peer certificate cannot be authenticated with given CA certificates, SSL certificate problem: unable to get local issuer certificate

When reading the certificate from https://dl.broadcom.com, you see it is not signed by a public Certificate Authority:

# echo | openssl s_client -connect dl.broadcom.com:443 2>/dev/null -showcerts | sed -n '/^-----BEGIN CERTIFICATE-----/,/^-----END CERTIFICATE-----/p' | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout

subject=/C=US/ST=California/L=Palo Alto/O=VMware, Inc./OU=IT Operations/CN=*.vmware.com
issuer=/C=US/O=acme/OU=Lab/CN=proxy.acme.com

subject=/C=US//O=acme/OU=Lab/CN=proxy.acme.com
issuer=/DC=com/DC=acme/DC=ad/CN=CA-INTERM1

subject=/DC=com/DC=acme/DC=ad/CN=CA-INTERM1
issuer=/CN=CA-ROOT

Environment

VMware vCenter Server 8.0x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

This can happen if there is a transparent proxy performing SSL Interception for outgoing HTTPS traffic. The certificate is being replaced by one that is signed by a Certificate Authority that is not trusted by the appliance.

Resolution

To prevent this issue, ensure that communication between the vCenter Server and https://dl.broadcom.com is not decrypted and re-encrypted by the SSL introspection software, but can instead pass untouched.
Should this not be an option, the issue can be worked around by applying the steps in the section "Workaround" below.

Workaround:
To work around this problem, please apply the following steps:

1. Download the certificate chain as presented by the proxy, and split them into individual files:

   # echo | openssl s_client -connect dl.broadcom.com:443 2>/dev/null -showcerts | sed -n '/^-----BEGIN CERTIFICATE-----/,/^-----END CERTIFICATE-----/p' | csplit -z -f /tmp/proxy-cert- -b%02d.crt /dev/stdin "/-----BEGIN CERTIFICATE-----/" "{*}"

   Note: If the proxy does not present the full CA chain, you will need to engage your PKI administration team to obtain the Certificate Authority certs that signed this certificate. 
2. Add the CA certificates to a file:

   # for file in $(ls /tmp/proxy-cert-* | grep -v proxy-cert-00); do cat $file >> /tmp/proxy-cas.pem; done

3. Publish the CA certificates to VMware Directory and refresh the TRUSTED_ROOTS store in VECS (certificates in the TRUSTED_ROOTS store are also saved to /etc/ssl/certs):

   # /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert /tmp/proxy-cas.pem
   # /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

4. If the vCenter is 7.0 U1c or later, the CA certificates will also need to be added to the PEM file that python references:

   # cp /usr/lib/python3.7/site-packages/certifi/cacert.pem /usr/lib/python3.7/site-packages/certifi/cacert.pem.backup
   # cat /tmp/proxy-cas.pem >> /usr/lib/python3.7/site-packages/certifi/cacert.pem

5. Restart the vSphere Update Manager service:

   # service-control --restart vmware-updatemgr