Creating Private Registry fails due to Private Registry CA Certificate Error - x509: Certificate signed by unknown authority

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When trying to create a private registry as per Deploying Supervisor Services from a Private Container Image Registry

It can fail with error on the supervisor like below

Configured Core Supervisor Services
Service: tkg.vsphere.vmware.com. Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://<registry-fqdn>/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority.

 

Environment

vSphere 8.0 U3

Cause

When adding the registry URL, system uses the hostname in the private registry URL to create directory for certificate sync. The issue is caused when the private registry URL being provided includes protocol or path,  hostname is not retrieved properly which leads to wrong directory name being used. Example like below. Directories are listed on Supervisor control plane nodes.

/etc/containerd/certs.d
/etc/containerd/.private-registries

 

   root@4<> [ ~ ]# ls -lR /etc/containerd/certs.d
    /etc/containerd/certs.d:
 
    '/etc/containerd/certs.d/https:':
    total 4
    drwxr-xr-x 2 root root 4096 Mar 5 10:20 <registry fqdn>

    '/etc/containerd/certs.d/https:/<registry fqdn>':
    total 4
    -rw------- 1 root root 1256 Mar 5 10:20 ca.crt

    '/etc/containerd/.private-registries/https:':
    total 4
    drwxr-xr-x 2 root root 4096 Mar 5 10:20 <registry fqdn>

    '/etc/containerd/.private-registries/https:/<registry fqdn>

Resolution

1. Remove the existing registry
2. Make sure folders containing '<registry fqdn>' are removed from below directories
a) /etc/containerd/.private-registries/
b) /etc/containerd/certs.d/
3. Re-add private registry with below configuration
a) Use '<registry fqdn>' ONLY in the Registry host URL. (Note, with NO "https://")
b) Use the cert included in the harbor.<registry fqdn>.crt in TLS Certificate