ESXi Compontents "Synchronize Now" in Fleet manager task, fails with error message "Failed to execute UMDS command: /opt/vmware/vmware-umds/bin/vmware-umds -D -m --info-level error --proxy-ip <proxy> --proxy-port <port> --task-id #####

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

In a VCF Environment , where SDDC is set up as a UMDS [see documentation] the Synchronize Now task under [path below], fails.
-> Fleet Management -> Vcf Instances -> Vcf Instance {TBD} -> Binary Management -> ESXi Components "Synchronize Now"

Error message

Message: Failed to execute UMDS command: /opt/vmware/vmware-umds/bin/vmware-umds -D -m --info-level error --proxy-ip <Proxy> --proxy-port <Port> --task-id ##########-##########-##########-##########
Remediation Message:
Reference Token:

Logs show the following

─$ grep -i vmware-downloadService /var/log/VMware/vcf/lcm/lcm-debug.log

YYYY-MM-DDTHH:MM:SS INFO  [vcf_lcm,6997059bbd2f00115930d0c4c78bc040,8e3f] [c.v.e.s.l.s.u.i.UmdsDownloadServiceImpl,pool-11-thread-6] YYYY-MM-DDTHH:MM:SS.335Z error vmware-downloadService[583352] [Originator@6876 sub=httpDownload] [httpDownloadPosix 782] curl_easy_perform() failed: cURL Error: SSL peer certificate or SSH remote key was not OK, SSL certificate problem: unable to get local issuer certificate
YYYY-MM-DDTHH:MM:SS INFO  [vcf_lcm,6997059bbd2f00115930d0c4c78bc040,8e3f] [c.v.e.s.l.s.u.i.UmdsDownloadServiceImpl,pool-11-thread-6] YYYY-MM-DDTHH:MM:SS.337Z error vmware-downloadService[583352] [Originator@6876 sub=httpDownload] [httpDownloadPosix 782] [backtrace begin] product: VMware vSphere Update Manager Download Service, version: 9.0.2, build: build-25148086, tag: vmware-downloadService, cpu: x86_64, os: linux, buildType: release
YYYY-MM-DDTHH:MM:SS INFO  [vcf_lcm,6997059bbd2f00115930d0c4c78bc040,8e3f] [c.v.e.s.l.s.u.i.UmdsDownloadServiceImpl,pool-11-thread-6] YYYY-MM-DDTHH:MM:SS.338Z error vmware-downloadService[583352] [Originator@6876 sub=HostUpdateDepotManager] [patchDepotManager 2272] Access token couldn't be received from VVS server's auth endpoint.
YYYY-MM-DDTHH:MM:SS INFO  [vcf_lcm,6997059bbd2f00115930d0c4c78bc040,8e3f] [c.v.e.s.l.s.u.i.UmdsDownloadServiceImpl,pool-11-thread-6] YYYY-MM-DDTHH:MM:SS.338Z error vmware-downloadService[583352] [Originator@6876 sub=Default] [hostUpdate20Downloader 333] VVS download failed. Message: Access token couldn't be received from VVS server's auth endpoint.
LocalProcess INFO: YYYY-MM-DDTHH:MM:SS - YYYY-MM-DDTHH:MM:SS error vmware-downloadService[583352] [Originator@6876 sub=httpDownload] [httpDownloadPosix 782] curl_easy_perform() failed: cURL Error: SSL peer certificate or SSH remote key was not OK, SSL certificate problem: unable to get local issuer certificate
LocalProcess INFO: YYYY-MM-DDTHH:MM:SS - YYYY-MM-DDTHH:MM:SS error vmware-downloadService[583352] [Originator@6876 sub=httpDownload] [httpDownloadPosix 782] [backtrace begin] product: VMware vSphere Update Manager Download Service, version: 9.0.2, build: build-25148086, tag: vmware-downloadService, cpu: x86_64, os: linux, buildType: release
LocalProcess INFO: YYYY-MM-DDTHH:MM:SS - YYYY-MM-DDTHH:MM:SS error vmware-downloadService[583352] [Originator@6876 sub=HostUpdateDepotManager] [patchDepotManager 2272] Access token couldn't be received from VVS server's auth endpoint.
LocalProcess INFO: YYYY-MM-DDTHH:MM:SS - YYYY-MM-DDTHH:MM:SS error vmware-downloadService[583352] [Originator@6876 sub=Default] [hostUpdate20Downloader 333] VVS download failed. Message: Access token couldn't be received from VVS server's auth endpoint.

Environment

VCF 9

Cause

The connection from SDDC to the public URLs is intercepted via SSL introspection.
In this specific case, it's show in the following error - VVS download failed

LocalProcess INFO: <Date && Time> - <Date && Time> error vmware-downloadService[583352] [Originator@6876 sub=Default] [hostUpdate20Downloader 333] VVS download failed. Message: Access token couldn't be received from VVS server's auth endpoint.

Where VVS server is auth.esp.vmware.com
Complete list for the mandatory whitelisted URLS is here https://knowledge.broadcom.com/external/article/327186/public-url-list-for-sddc-manager.html

Resolution

Verify which URL is intercepted, in this case it is "auth.esp.vmware.com", as the error references VVS

1. SSH on the SDDC via vcf

2. Change to Root via
# su -3. Get the certificate via openssl command

# openssl s_client -connect auth.esp.vmware.com:443 -proxy <proxy:port> -servername auth.esp.vmware.com -showcerts </dev/null 2>/dev/null | openssl x509 -outform PEM > /home/vcf/auth.pem

4. Read the exported certificate.
In case of interception, in the ISSUER field, we see CUstom Certificate Authority references - i.e. <CU CA>

# openssl x509 -in /home/vcf/auth.pem -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:85:67:19:57:2a:54:4c:e8:23:09:34:4e:f1:2b:50:d6:3a:9b:73
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = <CU CA>, O = <CU CA>, OU = <CU CA>, CN = <CU CA>
        Validity
            Not Before: Nov 27 00:00:00 2025 GMT
            Not After : Apr 14 23:59:59 2026 GMT
        Subject: C = US, ST = California, L = Palo Alto, O = Broadcom Inc., CN = auth.esp.vmware.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
...

5. As an example, we expect the following values under ISSUER, where no interception is present

# openssl x509 -in /home/vcf/auth.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            08:1b:e4:18:b4:f3:2e:8f:ac:d7:c5:11:bd:a5:63:7a
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
        Validity
            Not Before: Nov 27 00:00:00 2025 GMT
            Not After : Apr 14 23:59:59 2026 GMT
        Subject: C = US, ST = California, L = Palo Alto, O = Broadcom Inc., CN = auth.esp.vmware.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:

6. Reach out to your Network team, and make sure the following URLs Public URL list for VCF Products are Whitelisted.

7. Manually trigger this task by following the path Fleet Management > Lifecycle -> Our VCF Instance > Binary Management > ESXi Components > Synchronize now.

Workaround: Update the Photon OS trust store on SDDC Manager to trust the proxy CA certificate chain.

Gracefully shutdown SDDC Manager and take a snapshot.
Power it back up and ssh with vcf account and then su to root.
Add each individual base64 formatted CA pem file (obtain using openssl command above or from browser to dl.broadcom.com) in the cert chain of the proxy performing ssl interception to the OS CA store: /etc/ssl/certs
Run /usr/bin/rehash_ca_certificates.sh which updates /etc/pki/tls/certs/ca-bundle.crt
Verify on SDDC Manager curl https://dl.broadcom.com now returns, "invalid token" instead of a ssl error.
Reattempt to trigger the ESX download task.

Additional Information

Offline Depot enterprise certificate store issues when downloading ESXI Components within the SDDC Manager and operations.