vSphere Update Manager / Lifecycle Manager fails to download updates due to SSL Interception

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
Attempting to download patch definitions in the vSphere Web Client or vSphere Client (Flash) fails with the following symptoms:

vSphere Web Client (HTML5):

• When looking in Home > Update Manager > [vum-server]  > Manage > Download Settings in vCenter Server 6.5/6.7 or in Home > Lifecycle Manager > Settings > Patch Setup in vCenter Server 7.0.x, the download configured for https://hostupdate.vmware.com appear as "Not Connected"
• After clicking the Download Now button, the task fails with a message similar to:

  Web sites: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml; https://hostupdate.vmware.com/software/VUM/PRODUCTION/csco-main/csco-depot-index.xml; hosting the patch definitions and patches cannot be accessed or have no patch data. Check the Internet connectivity

vSphere Client (Flash):

• When looking in Home > Update Manager > Settings > Administration Settings > Patch Setup, the download sources configured for https://hostupdate.vmware.com are in the status "Not Connected"
• Going to Home > Update Manager > Settings > Administration Settings > Patch Downloads and clicking Download Now will result in a failed task with a message similar to:

  Web sites: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml; hosting the patch definitions and patches cannot be accessed or have no patch data. Check the Internet connectivity

vCenter (BASH) shell:

• Attempting to pull down the XML file with curl results in an error:

  # curl https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
  
  curl: (60) SSL certificate problem: unable to get local issuer certificate
  More details here: https://curl.haxx.se/docs/sslcerts.html
  
  curl failed to verify the legitimacy of the server and therefore could not
  establish a secure connection to it. To learn more about this situation and
  how to fix it, please visit the web page mentioned above.

In /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log you find the following entries:

[2019-09-20 21:00:11:185 'httpDownload' 140052601693952 ERROR]  [httpDownloadPosix, 649] curl_easy_perform() failed: cURL Error: Peer certificate cannot be authenticated with given CA certificates, SSL certificate problem: unable to get local issuer certificate
[2019-09-20 21:00:11:185 'DownloadMgr' 140052601693952 ERROR]  [downloadMgr, 629] Executing download job {140052105550256} throws error: curl_easy_perform() failed: cURL Error: Peer certificate cannot be authenticated with given CA certificates, SSL certificate problem: unable to get local issuer certificate

When reading the certificate from https://hostupdate.vmware.com, you see it is not signed by a public Certificate Authority:

# echo | openssl s_client -connect hostupdate.vmware.com:443 2>/dev/null -showcerts | sed -n '/^-----BEGIN CERTIFICATE-----/,/^-----END CERTIFICATE-----/p' | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout

subject=/C=US/ST=California/L=Palo Alto/O=VMware, Inc./OU=IT Operations/CN=*.vmware.com
issuer=/C=US/O=acme/OU=Lab/CN=proxy.acme.com

subject=/C=US//O=acme/OU=Lab/CN=proxy.acme.com
issuer=/DC=com/DC=acme/DC=ad/CN=CA-INTERM1

subject=/DC=com/DC=acme/DC=ad/CN=CA-INTERM1
issuer=/CN=CA-ROOT

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

This can happen if there is a transparent proxy performing SSL Interception for outgoing HTTPS traffic. The certificate is being replaced by one that is signed by a Certificate Authority that is not trusted by the appliance.

Resolution

To prevent this issue, ensure that communication between the vCenter Server and https://hostupdate.vmware.com is not decrypted and re-encrypted by the SSL introspection software, but can instead pass untouched.
Should this not be an option, the issue can be worked around by applying the steps in the section "Workaround" below.

Workaround:
To work around this problem, please apply the following steps:

1. Download the certificate chain as presented by the proxy, and split them into individual files:

   # echo | openssl s_client -connect hostupdate.vmware.com:443 2>/dev/null -showcerts | sed -n '/^-----BEGIN CERTIFICATE-----/,/^-----END CERTIFICATE-----/p' | csplit -z -f /tmp/proxy-cert- -b%02d.crt /dev/stdin "/-----BEGIN CERTIFICATE-----/" "{*}"

   Note: If the proxy does not present the full CA chain, you will need to engage your PKI administration team to obtain the Certificate Authority certs that signed this certificate. 
2. Add the CA certificates to a file:

   # for file in $(ls /tmp/proxy-cert-* | grep -v proxy-cert-00); do cat $file >> /tmp/proxy-cas.pem; done

3. Publish the CA certificates to VMware Directory and refresh the TRUSTED_ROOTS store in VECS (certificates in the TRUSTED_ROOTS store are also saved to /etc/ssl/certs):

   # /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert /tmp/proxy-cas.pem
   # /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

4. If the vCenter is 7.0 U1c or later, the CA certificates will also need to be added to the PEM file that python references:

   # cp /usr/lib/python3.7/site-packages/certifi/cacert.pem /usr/lib/python3.7/site-packages/certifi/cacert.pem.backup
   # cat /tmp/proxy-cas.pem >> /usr/lib/python3.7/site-packages/certifi/cacert.pem

5. Restart the vSphere Update Manager service:

   # service-control --restart vmware-updatemgr