VCF 9.1 Software Depot fails with error, "Failed to connect to the authorization server to obtain access token." when using a SSL Inspection proxy

Products

• VMware Cloud Foundation
• VCF Operations

Issue/Introduction

VCF Operations > Software Depot Wizard shows the following error:

Failed to connect to authorization server" or "Failed to connect to Broadcom OAuth Authorization server

 

 

A similar error is observed in the FDS pod log (depot-service):

javax.net.ssl.SSLHandshakeException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target

 

Environment

• VMware Cloud Foundation 9.1
• VCF Operations 9.1
• VCF Software Depot 9.1 

Cause

In network environments with SSL Inspection enabled (a security firewall or proxy performing TLS/HTTPS interception), the firewall intercepts outbound connections to online VCF software depot URLs (e.g. dl.broadcom.com, eapi.broadcom.com) and re-encrypts the traffic using its own CA-signed certificate instead of the real Broadcom certificate. Since the Software Depot trust store does not contain the firewall's Root CA/Intermediate CA, the TLS handshake fails and the online depot cannot be configured or used.

Resolution

Follow the below mentioned steps to collect and add the TLS certificates into the Fleet Depot Service's trust store:

1. Download the add_trusted_certificate.py attached to this KB.
2. Copy the add_trusted_certificate.py to the VCF SDDC Manager /home/vcf/ directory.
3. SSH into the VCF SDDC Manager with the vcf user and then elevate to root with su.
4. Run the add_trusted_certificate.py script ( When prompted provide the [e-mail] password )

   • Option 1: Fetch the proxy cert and trust the certificates:

     python3 add_trusted_certificate.py --vsp-fqdn <vsp-fqdn> --fetch-from-proxy proxy.corp.example.com:8080 --connect-host eapi.broadcom.com:443

   • Option 2: Manually provide a .pem file:

     python3 add_trusted_certificate.py --vsp-fqdn <vsp-fqdn> --cert-file /tmp/firewall-root-ca.pem

   • Note: <vsp-fqdn> is the VMSP Platform / VCF Services Runtime hostname.
5. Retry the Depot connection workflow.
6. The Software Depot UI will go blank. Wait for ~15 minutes for services to start.

Additional Information

• Ensure to provide the full certificate chain when choosing Option 2 in the Step 4 of the resolution section. Following is the example of how the certificate chain should be:

  leaf certificate       <------   ( skip if not there )
  intermediate CA        <------   ( mandatory )
  root CA                <------  ( mandatory )

• Verify that all of the required URLs are allowed. Refer: Public URLs Required for Online Functionalities for VCF and vSphere Foundation
• Verify all IPs Control Plane and Worker node IPs are allowed on the Proxy Server for INGRESS Refer: Activation of an Online Depot fails with the error "Error occurred on the authorization server. Please try again"