VCF 9.1 with Online Connected depot: Task Downloading Esx metadata, vibs and vendor add-ons is failing due to missing proxy certificate

Products

• VMware Cloud Foundation
• VCF Operations

Issue/Introduction

• Online environment where the Depot is connected.
  
  
• Task "Downloading Esx metadata, vibs and vendor add-ons" task is failing with error:
  
  Failed to execute UMDS command: /opt/vmware/vmware-umds/bin/vmware-umds -D --info-level error --task-id ###########
  
  
  
  
  
  

• Connection to vvs.broadcom.com seems to be failing:

  yyyy-mm-ddThh:mm:ss.761739946Z stdout F yyyy/mm/dd hh:mm:ss Received request: GET /v1/products/bundles/vlcm-compatibility-data/v1/vlcm-interop-vcg-bundle?format=gz
  yyyy-mm-ddThh:mm:ss.761756957Z stdout F yyyy/mm/dd hh:mm:ss Forwarding request to: https://vvs.broadcom.com/v1/products/bundles/vlcm-compatibility-data/v1/vlcm-interop-vcg-bundle (X-Forwarded-For: <IP_ADRRESS>, <IP_ADRRESS>)
  yyyy-mm-ddThh:mm:ss.837667616Z stdout F yyyy/mm/dd hh:mm:ss Request failed on attempt 1 for GET https://vvs.broadcom.com/v1/products/bundles/vlcm-compatibility-data/v1/vlcm-interop-vcg-bundle?format=gz: 
  Get "https://vvs.broadcom.com/v1/products/bundles/vlcm-compatibility-data/v1/vlcm-interop-vcg-bundle?format=gz": tls: failed to verify certificate: x509: certificate signed by unknown authority
  yyyy-mm-ddThh:mm:ss.837707996Z stdout F yyyy/mm/dd hh:mm:ss SSL_ERROR: Detected SSL/TLS certificate error, will attempt certificate reload
  yyyy-mm-ddThh:mm:ss.837717076Z stdout F yyyy/mm/dd hh:mm:ss SSL_ERROR: Reloading certificates and retrying (attempt 1/1)
  yyyy-mm-ddThh:mm:ss.837724126Z stdout F yyyy/mm/dd hh:mm:ss CERT_RELOAD: Starting certificate reload...
  yyyy-mm-ddThh:mm:ss.840355565Z stdout F yyyy/mm/dd hh:mm:ss Loaded platform trust store from /etc/platform/trust/bundle.pem
  yyyy-mm-ddThh:mm:ss.840404976Z stdout F yyyy/mm/dd hh:mm:ss Depot certificate file at /etc/depot/trust/certificates.pem is empty (skipping)
  yyyy-mm-ddThh:mm:ss.840419586Z stdout F yyyy/mm/dd hh:mm:ss Proceeding without proxy (DISABLE_PROXY=false, PROXY_HOST=, PROXY_PORT=, PROXY_SCHEME=http).
  yyyy-mm-ddThh:mm:ss.840431356Z stdout F yyyy/mm/dd hh:mm:ss CERT_RELOAD: Successfully reloaded certificates and recreated HTTP client
  yyyy-mm-ddThh:mm:ss.840442196Z stdout F yyyy/mm/dd hh:mm:ss SSL_ERROR: Certificates reloaded successfully, retrying request...
  yyyy-mm-ddThh:mm:ss.978947866Z stdout F yyyy/mm/dd hh:mm:ss Request failed on attempt 2 for GET https://vvs.broadcom.com/v1/products/bundles/vlcm-compatibility-data/v1/vlcm-interop-vcg-bundle?format=gz: 
  Get "https://vvs.broadcom.com/v1/products/bundles/vlcm-compatibility-data/v1/vlcm-interop-vcg-bundle?format=gz": tls: failed to verify certificate: x509: certificate signed by unknown authority
  yyyy-mm-ddThh:mm:ss.978983136Z stdout F yyyy/mm/dd hh:mm:ss SSL_ERROR: Detected SSL/TLS certificate error, will attempt certificate reload
  yyyy-mm-ddThh:mm:ss.978989486Z stdout F yyyy/mm/dd hh:mm:ss SSL_ERROR: Max retry attempts (1) exhausted, giving up

Environment

• VMware Cloud Foundation 9.1
• VCF Software Depot 9.1 

Cause

If your network uses a proxy or firewall with SSL/TLS interception, outbound traffic to VCF software depot URLs (like vvs.broadcom) gets intercepted and re-encrypted.
The security device uses its own certificate rather than the official Broadcom certificate. Consequently, the TLS connection fails because the firewall's Root/Intermediate CA is missing from the Software Depot trust store.

Resolution

Use the resolution steps of the following KB article to add TLS certificates into the Fleet Depot Service's Trust Store:

VCF 9.1 Software Depot fails with error, "Failed to connect to the authorization server to obtain access token." when using a SSL Inspection proxy

Additional Information

Verify that all of the required URLs are allowed:

Public URLs Required for Online Functionalities for VCF and vSphere Foundation

Verify all IPs Control Plane and Worker node IPs are allowed on the Proxy Server for INGRESS:

Activation of an Online Depot fails with the error "Error occurred on the authorization server. Please try again"