Offline Depot enterprise certificate store issues when downloading ESXI Components within the SDDC Manager and operations.

Products

• VMware SDDC Manager

Issue/Introduction

Attempting to synchronize ESXi Components in SDDC Manager fails.

 

The SDDC and fleet manager lifecycle are configured with an offline depot. 

 

The fleet manager operations will accept the certificate of the offline depot. 

 

From VMware Cloud Foundation Operations->Fleet Management->Lifecycle

Under the VCF Instances

Click on an instance

ESXi Components 

Synchronize Now 

 

This will cause SDDC Manager to fail to sync the esx updates

Environment

VMware SDDC Manager 9.0

Cause

The synchronize now feature triggers SDDC Manager to kick off the embedded UMDS task to synchronize the updates from the offline repo.  

If the repo is over https it could fail because SDDC Manager does not trust the certificate.

This is due to curl command being used which utilizes the OS CA store.

Reviewing the log /var/log/vmware/vmware-updatemgr/umds/vmware-downloadService.log on SDDC Manager 
YYYY-MM-DDTHH:MM:SS.388Z error vmware-downloadService[959145] [Originator@6876 sub=httpDownload] [httpDownloadPosix 782] [backtrace begin] product: VMware vSphere Update Manager Download Service, version: 9.0.0, build: build-24695687, tag: vmware-downloadService, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0048395D]
--> backtrace[01] libvmacore.so[0x003730D8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x003855E5]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x00E463AF]
--> backtrace[04] libvci-vcIntegrity.so[0x00E4681F]
--> backtrace[05] libvci-vcIntegrity.so[0x00E46D19]: Sysimage::HttpDownloadFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Integrity::ProxyServer const&, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool const&)
--> backtrace[06] libvci-vcIntegrity.so[0x00E41B61]: Sysimage::DownloadJobHandler::Download()
--> backtrace[07] libvmacore.so[0x002CFC04]
--> backtrace[08] libvmacore.so[0x002D550F]
--> backtrace[09] libvmacore.so[0x00462AEB]
--> backtrace[10] libc.so.6[0x000890C4]
--> backtrace[11] libc.so.6[0x0010916C]
--> backtrace[12] (no module)
--> [backtrace end]
YYYY-MM-DDTHH:MM:SS.388Z verbose vmware-downloadService[959145] [Originator@6876 sub=httpDownload] [httpDownloadPosix 756] Cleanup SSL context
YYYY-MM-DDTHH:MM:SS.388Z error vmware-downloadService[959145] [Originator@6876 sub=DownloadMgr] [downloadMgr 709] Executing download job {139698576042880} throws error: curl_easy_perform() failed: cURL Error: SSL peer certificate or SSH remote key was not OK, SSL certificate problem: self-signed certificate in certificate chain
YYYY-MM-DDTHH:MM:SS.388Z error vmware-downloadService[959154] [Originator@6876 sub=Default] [updateDownloaderImpl 116] File download error: curl_easy_perform() failed: cURL Error: SSL peer certificate or SSH remote key was not OK, SSL certificate problem: self-signed certificate in certificate chain
YYYY-MM-DDTHH:MM:SS.388Z error vmware-downloadService[959154] [Originator@6876 sub=Default] [updateDownloaderImpl 245] failed to download vendor index file: https://###.###.###:443/umds-patch-store/hostupdate/__hostupdate20-consolidated-index__.xml

Resolution

Steps to update the the Photon OS CA certificates.

Add CA certificates for the Offline repository to the OS CA store, /etc/ssl/certs

Run /usr/bin/rehash_ca_certificates.sh which updates /etc/pki/tls/certs/ca-bundle.crt