Installing or Upgrading VKS Supervisor service fails with error "tls: failed to verify certificate: x509: certificate signed by unknown authority"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• VKS Service installation or upgrade is failing with the error below.
  
  Configured Core Supervisor Services
Service: tkg.vsphere.vmware.com. Reason: ReconcileFailed. Message:
vendir: Error: Syncing directory '0': Syncing directory '.' with
imgpkgBundle contents: Fetching image: Error while preparing a
transport to talk with the registry: Unable to create round tripper: Get
"https://projects.packages.broadcom.com/v2/": tls: failed to verify
certificate: x509: certificate signed by unknown authority .
Service: velero.vsphere.vmware.com. Status: Running


• On running the below command to obtain the certificate presented by projects.packages.broadcom.com, the certificate obtained is a custom certificate and not the one signed by DigiCert TLS CA1.
  
  openssl s_client -connect projects.packages.broadcom.com:443

Environment

• VMware vSphere Kubernetes Service
• VMware vCenter Server 8.0U3

Cause

The issue stems from SSL offloading at the intermediary layer, specifically within the firewall, load balancer, or proxy configurations.

Resolution

Note: The resolution is only applicable for 8.0U3 and later.

Register "projects.packages.broadcom.com" as a private registry with a custom certificate (the CA chain that the firewall replaces with) - allowing the supervisor to trust anything from this registry.

Refer to the Install and Use the Supervisor Service guide for detailed instructions on adding a private registry and completing the service registration.

Additional Information

Exclude the IPs for the projects.packages.broadcom.com from SSL termination. However these are subject to change from time to time.