Lifecycle Manager fails with "The download source dl.broadcom.com is invalid or cannot be reached now." due to ssl interception

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
Adding url to download patch definitions in the vSphere Web Client fails with the following symptoms:

vSphere Web Client (HTML5):

When adding url to download patches in Home > Update Manager > [vum-server] > Manage > Download Settings in vCenter Server 6.5/6.7 or in Home > Lifecycle Manager > Settings > Patch Setup in vCenter Server 7.x/8.x
After clicking the New button, entering the url with token (https://dl.broadcom.com/<downloadToken>/PROD/COMP/VCENTER/...) it fails with a message when clicking on save with:
```
The download source https://dl.broadcom.com/<Download Token>/PROD/COMP/VCENTER/... is invalid or cannot be reached now.
```

vCenter (BASH) shell:

Attempting to connect with curl results in an error:

# curl https://dl.broadcom.com/<Download Token>//PROD/COMP/VCENTER/...

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://xxxx.xxxx.xx/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

In /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log you find the following entries:

[2019-09-20 21:00:11:185 'httpDownload' 140052601693952 ERROR]  [httpDownloadPosix, 649] curl_easy_perform() failed: cURL Error: Peer certificate cannot be authenticated with given CA certificates, SSL certificate problem: unable to get local issuer certificate
[2019-09-20 21:00:11:185 'DownloadMgr' 140052601693952 ERROR]  [downloadMgr, 629] Executing download job {140052105550256} throws error: curl_easy_perform() failed: cURL Error: Peer certificate cannot be authenticated with given CA certificates, SSL certificate problem: unable to get local issuer certificate

When reading the certificate from https://dl.broadcom.com, you see it is not signed by a public Certificate Authority:

# echo | openssl s_client -connect dl.broadcom.com:443 2>/dev/null -showcerts | sed -n '/^-----BEGIN CERTIFICATE-----/,/^-----END CERTIFICATE-----/p' | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout

subject=/C=US/ST=California/L=Palo Alto/O=VMware, Inc./OU=IT Operations/CN=*.vmware.com
issuer=/C=US/O=example/OU=Lab/CN=proxy.example.com

subject=/C=US/O=example/OU=Lab/CN=proxy.example.com
issuer=/DC=com/DC=example/DC=ad/CN=CA-INTERM1

subject=/DC=com/DC=example/DC=ad/CN=CA-INTERM1
issuer=/CN=CA-ROOT

Environment

VMware vCenter Server 8.0x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

This can happen if there is a transparent proxy performing SSL Interception for outgoing HTTPS traffic. The certificate is being replaced by one that is signed by a Certificate Authority that is not trusted by the appliance.

Resolution

To prevent this issue, ensure that communication between the vCenter Server and https://dl.broadcom.com is not decrypted and re-encrypted by the SSL introspection software, but can instead pass untouched.

Should this not be an option, the issue can be worked around by applying the steps in the "Workaround" sections below.

Workaround A:

Obtain the full signing chain employed by the proxy from your internal proxy administrator(s) in base64 x509 format in a single file in order with the root at the bottom and proxy signing cert at the top.
Copy this file to /tmp/proxy-cas.pem on the vCenter server.
Publish the CA certificates to VMware Directory and refresh the TRUSTED_ROOTS store in VECS (certificates in the TRUSTED_ROOTS store are also saved to /etc/ssl/certs):
```
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert /tmp/proxy-cas.pem
# /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
```

If the vCenter is 7.0 U1c or later, the CA certificates will also need to be added to the PEM file that python references:

# cp /usr/lib/python3.7/site-packages/certifi/cacert.pem /usr/lib/python3.7/site-packages/certifi/cacert.pem.backup
# cat /tmp/proxy-cas.pem >> /usr/lib/python3.7/site-packages/certifi/cacert.pem

Restart the vSphere Update Manager service:

# service-control --restart vmware-updatemgr

Workaround B:

Note: If the proxy does not present the full CA chain, you will need to engage your PKI administration team to obtain the Certificate Authority certs that signed this certificate. See above

Download the certificate chain as presented by the proxy, and split them into individual files:

# echo | openssl s_client -connect dl.broadcom.com:443 2>/dev/null -showcerts | sed -n '/^-----BEGIN CERTIFICATE-----/,/^-----END CERTIFICATE-----/p' | csplit -z -f /tmp/proxy-cert- -b%02d.crt /dev/stdin "/-----BEGIN CERTIFICATE-----/" "{*}"

Add the CA certificates to a file:

# for file in $(ls /tmp/proxy-cert-* | grep -v proxy-cert-00); do cat $file >> /tmp/proxy-cas.pem; done

Publish the CA certificates to VMware Directory and refresh the TRUSTED_ROOTS store in VECS (certificates in the TRUSTED_ROOTS store are also saved to /etc/ssl/certs):
```
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert /tmp/proxy-cas.pem
# /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
```

If the vCenter is 7.0 U1c or later, the CA certificates will also need to be added to the PEM file that python references:

# cp /usr/lib/python3.7/site-packages/certifi/cacert.pem /usr/lib/python3.7/site-packages/certifi/cacert.pem.backup
# cat /tmp/proxy-cas.pem >> /usr/lib/python3.7/site-packages/certifi/cacert.pem

Restart the vSphere Update Manager service:

# service-control --restart vmware-updatemgr