Failed to upgrade TKG service due to self-signed certificate used by standalone Harbor

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

- A standalone harbar with self-signed certificate was add to the container registry of a supervisor cluster.

- When upgrading the TKG service, the task failed due to "x509: certificate signed by unknown authority"

- The output of "kubectl describe pkgi XXXX" is similar to:

  usefulErrorMessage: |
    vendir: Error: Syncing directory '0':
      Syncing directory '.' with imgpkgBundle contents:
        Fetching image:
          Error while preparing a transport to talk with the registry:
            Unable to create round tripper:
              Get "https://<harbor_fqdn>/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
  version: 3.2.0

Environment

vSphere with Tanzu 8.x

Cause

- When upgrading the TKG Service, the CPVM doesn't identify CA cert correctly, so it is not able to pull image from harbor

service-control --stop --all
service-control --start --all

Resolution

1. Make sure the CA certificate which is used to sign harbor's certificate is correct.

2. Make sure you add the harbor fqdn without 'https://'

https://knowledge.broadcom.com/external/article/390454/creating-private-registry-fails-due-to-p.html

3. SSH to each CPVM, make sure all CPVM have the same certificate under '/etc/containerd/certs.d' and '/etc/containerd/private-registries'

4. SSH to CPVM and check the CA certificate 

openssl x509 -in /etc/containerd/certs.d/<harbor_fqdn>/ca.crt -text -noout

5. Restart vCenter's all services

service-control --stop --all
service-control --start --all

6. If it's not working, restart CPVM one by one. 

 

Please contact Broadcom Support team if you need any concern