Failed to upgrade TKG service due to self-signed certificate used by standalone Harbor

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• A standalone harbor with self-signed certificate was add to the container registry of a supervisor cluster.
• When upgrading the TKG service, the task failed due to "x509: certificate signed by unknown authority"
• The output of "kubectl describe pkgi XXXX" is similar to:

  usefulErrorMessage: |
    vendir: Error: Syncing directory '0':
      Syncing directory '.' with imgpkgBundle contents:
        Fetching image:
          Error while preparing a transport to talk with the registry:
            Unable to create round tripper:
              Get "https://<harbor_fqdn>/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
  version: 3.2.0

Environment

vSphere with Tanzu 8.x

Cause

When upgrading the TKG Service, the CPVM doesn't identify CA cert correctly, so it is not able to pull image from harbor

Resolution

To resolve this issue, follow the below steps:

• Make sure the CA certificate which is used to sign harbor's certificate is correct.
• Make sure you add the harbor FDQN without 'https://'

Creating Private Registry fails due to Private Registry CA Certificate Error - x509: Certificate signed by unknown authority

• SSH to each CPVM, make sure all CPVM have the same certificate under '/etc/containerd/certs.d' and '/etc/containerd/private-registries'
• SSH to CPVM and check the CA certificate 

openssl x509 -in /etc/containerd/certs.d/<harbor_fqdn>/ca.crt -text -noout

• Restart vCenter's all services

service-control --stop --all
service-control --start --all

• If it's not working, restart CPVM one by one. 

Additional Information

Please engage Broadcom Support in case of any concern.