UMDS unable to download updates when SSL inspection is being used

Products

VMware vCenter Server 8.0 VMware vCenter Server 7.0 VMware vCenter Server

Issue/Introduction

UMDS is running in a secure environment where SSL inspection is being used, e.g. by an SSL inspection proxy.

This is a system that intercepts incoming SSL traffic, decrypts it for inspection purposes, and the re-encrypts the traffic with its own certificate before sending it on the the intended receiver.

In this setup, UMDS fails to download any patches from the official download repository

When running try to enforce the patch download using "vmware-umds -D" the following error stack can be seen in the output:

[YYYY-MM-DDTHH:MM:SS] verbose vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * CONNECT tunnel established, response 200
[YYYY-MM-DDTHH:MM:SS] verbose vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * ALPN: curl offers http/1.1
[YYYY-MM-DDTHH:MM:SS] verbose vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * CAfile: /etc/pki/tls/certs/ca-bundle.crt
[YYYY-MM-DDTHH:MM:SS] verbose vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * CApath: /etc/ssl/certs
[YYYY-MM-DDTHH:MM:SS] verbose vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * SSL certificate problem: unable to get local issuer certificate
[YYYY-MM-DDTHH:MM:SS] verbose vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * Closing connection
[YYYY-MM-DDTHH:MM:SS] error vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 685] curl_easy_perform() failed: CURL Error: SSL peer certificate or SSH remote key was not OK, SSL certificate problem: unable to get local issuer certificate
[YYYY-MM-DDTHH:MM:SS] error vmware-downloadService[09266] [Originator@6876 sub=httpDownload] [httpDownloadPosix 685] [backtrace begin] product: VMware vSphere Update Manager Download Service, version: 8.0.2, build: build-23319993, tag: vmware-downloadService, cpu: x86_64, os: linux, buildType: release
> backtrace[00] libvmacore.so[0x0053D3AB]
> backtrace[01] libvmacore.so[0x0043197E]: Vmacore :: System :: Stacktrace :: CaptureFullWork(unsigned int)
-> backtrace[02] libvmacore.so[0x00444252]: Vmacore :: System :: SystemFactory :: CreateBacktrace(Vmacore :: Ref<Vmacore :: System :: Backtrace>&)
-- > backtrace[03] libvci-vcIntegrity.so[0x00DA8C73]
>backtrace[04] libvci-vcIntegrity.so[0x00DA9074]
backtrace[05] libvci-vcIntegrity.so[0x00DA98B0]: Sysimage :: HttpDownloadFile(std ::_ cxx11 :: basic_string<char, std :: char_traits<char>, std :: allocator<char> > const&, std :: cxx11 :: basic_string<char, std :: char_traits<char>, std :: allocator<char> > const&, Integrity :: ProxyServer const&,
std ::_ cxx11 :: basic_string<wchar_t, std :: char_traits<wchar_t>, std :: allocator<wchar_t> > const&, int, int, int, std ::_ cxx11 :: basic_string<char, std :: char_traits<char>, std :: allocator<char> > const&, bool const&)
backtrace[06] libvci-vcIntegrity.so[0x00DA4CBA]: Sysimage :: DownloadJobHandler :: Download()
> backtrace[07] libvmacore.so[0x0037DBE6]
backtrace[08] libvmacore.so[0x003834F9]
-> backtrace[09] libvmacore.so[0x0051D093]
-- > backtrace[10] libc.so.6[0x00089C02]
-> backtrace[11] libc.so.6[0x0010EC40]
> backtrace[12] (no module)
-- > [backtrace end]
[YYYY-MM-DDTHH:MM:SS] error vmware-downloadService[09266] [Originator@6876 sub=DownloadMgr] [downloadMgr 701] Executing download job {140278933502000} throws error: curl_easy_perform() failed: CURL Error: SSL peer certificate or SSH remote key was not OK, SSL certificate problem: unable to get local issuer certificate

Environment

Update Manager Download Service

Cause

In order to establish a connection with the official download repository, UMDS must be able to authenticate the validity of the certificate the incoming traffic was encrypted with.

This is usually done by verifying that the trusted certificate chain for the certificate authority, who signed the certificate is in the trusted root certificate store of the operating system UMDS is installed on.

Resolution

Resolution:

To verify if you are experiencing the issue, try to connect to the download source with curl and compare the certificate information in the output with the ones in the example below, specifically for the issuer:

# curl -kv -x <proxy-URL> https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
(..)
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Broadcom Inc.; CN=hostupdate.vmware.com
*  start date: May  6 00:00:00 2024 GMT
*  expire date: May  7 23:59:59 2025 GMT
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1

Note: Most SSL inspection applications act as proxies - but curl does not automatically apply the proxy settings configured in the operating system, hence you will need to manually add the proxy in the command using the option -x as seen above.

If either the subject or the issuer in the output on your system are different from what can be seen above, please collect the trusted certificate chain, consisting of:

the certificate for the CA which is named as the issuer in the curl output,
any other intermediate CA which has signed the certificate in 1.
the root CA

Follow the applicable steps for your specific operating system to import this trusted certificate chain into the trust store of the OS.

These steps will vary depending on the type/distribution of operating system you are using. Please refer to the OS manual or reach out to the OS vendor in case you require assistance with this.