vSphere Kubernetes Cluster Packages Cannot Reconcile due to Private Registry CA Certificate Error - x509: Certificate signed by unknown authority

Products

VMware vSphere with Tanzu vSphere with Tanzu

Issue/Introduction

Packages are unable to Reconcile in a vSphere Kubernetes cluster.

While connected to the affected vSphere Kubernetes cluster's context, the following symptoms will be present:

Packages are in Reconcile failed state with the Useful Error Message below where my-private-registry.url will be the expected private registry URL used in the vSphere Kubernetes cluster:

```
kubectl get pkgi -A
```

kubectl describe pkgi <package name> -n <package namespace>

Useful Error Message:  vendir: Error: Syncing directory '0':
  Syncing directory '.' with imgpkgBundle contents:
    Imgpkg: exit status 1 (stderr: imgpkg: Error: Fetching image:
  Error while preparing a transport to talk with the registry:
    Unable to create round tripper:
      Get "https://my-private-registry.url/v2/": x509: certificate signed by unknown authority

In vSphere 7.X, the kapp-controller configmap object within the vSphere Kubernetes cluster does not contain the private registry's certificates.

In vSphere 8.0u2 and lower 8.X versions, while connected to the Supervisor cluster's context, the following symptoms will be present:

The KappControllerConfig configmap object for the affected vSphere Kubernetes cluster does not contain the private registry's certificates.

In vSphere 8.0u3 and higher, a private container registry has not been set up with the Supervisor cluster.

Environment

VMware vSphere 8.0

VMware vSphere 7.0 with Tanzu

This issue can occur on vSphere Kubernetes cluster regardless of whether or not it is managed by Tanzu Mission Control (TMC)

Starting in vSphere 8.0u3, a private container registry can be set up with the Supervisor cluster.

Cause

In vSphere 7.X, kapp-controller is manually installed within the vSphere Kubernetes cluster. The private registry's certificates will need to be added to the configmap of the kapp-controller within the vSphere Kubernetes cluster.

In vSphere 8.X and higher, kapp-controller is automatically installed on the Supervisor and propagated into vSphere Kubernetes clusters running on a TKR for vSphere 8. The private registry's certificates will need to be added to the configuration of kapp controller under the KappControllerConfig configmap object in the Supervisor cluster context. Certificates set in the KappControllerConfig configmap object will be propagated to the vSphere Kubernetes clusters running on a TKR for vSphere 8.

In vSphere 8.0u3 and higher, a private container registry should be used instead of manually modifying the kapp-controller configuration objects.

Resolution

The private registry's certificates will need to be added to the corresponding kapp-controller configmap object according to whether the environment is on vSphere 7.X or vSphere 8.X.

In vSphere 8.0u3 and higher, a private container registry should be used instead of manually modifying the kapp-controller configuration objects.

For vSphere Kubernetes Clusters in vSphere 7.X:

Connect to the affected vSphere Kubernetes cluster's context.
Locate the kapp-controller configmap:
- ```
kubectl get configmap -A | grep -i kapp
```

Edit the kapp-controller configmap and add the private registry's certificates under data.caCerts:

kubectl edit configmap kapp-controller-config -n <kapp-controller-configmap namespace>

apiVersion: v1
kind: ConfigMap
metadata:
  name: kapp-controller-config
  namespace: tkg-system
  annotations:
    kapp.k14s.io/change-group: apps.kappctrl.k14s.io/kapp-controller-config
data:
  caCerts: |-
    -----BEGIN CERTIFICATE-----
    xxx
    -----END CERTIFICATE-----

Recreate the kapp-controller pod to pick up the certificates:

```
kubectl get deploy -A | grep kapp
```

kubectl rollout restart deploy kapp-controller -n <kapp-controller-namespace>

Confirm that packages are no longer reporting certificate signed by unknown authority errors:
- ```
kubectl get pkgi -A
```

For vSphere Kubernetes Clusters in vSphere 8.0u3 and higher:

Follow the below documentation for setting up a private container registry:
- https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/8-0/vsphere-supervisor-services-and-workloads-8-0/deploying-supervisor-services-from-a-private-container-image-registry.html

For vSphere Kubernetes Clusters in vSphere 8.0u2 and lower:

Connect to the Supervisor cluster's context.
Locate the KappControllerConfig configmap for the affected vSphere Kubernetes cluster:
- ```
kubectl get configmap -A | grep -i kapp
```

Edit the KappControllerConfig configmap where my-cluster is the name of the affected vSphere Kubernetes cluster and my-namespace is the namespace of the configmap to add the private registry's certificates under kappController.config.caCerts:

kubectl edit KappControllerConfig my-cluster-kapp-controller-package -n my-namespace

apiVersion: run.tanzu.vmware.com/v1alpha3
kind: KappControllerConfig
metadata:
  ...
  name: my-cluster-kapp-controller-package
  namespace: my-namespace
  ...
spec:
  kappController:
    config:
      caCerts: |-
        -----BEGIN CERTIFICATE-----
        xxx
        -----END CERTIFICATE-----

Recreate the kapp-controller pod to pick up the certificates:

```
kubectl get deploy -A | grep kapp
```

kubectl rollout restart deploy kapp-controller -n <kapp-controller-namespace>

Confirm that packages within the Supervisor cluster are not reporting certificate signed by unknown authority errors:
- ```
kubectl get pkgi -A
```
Connect to the affected vSphere Kubernetes cluster and confirm that packages within the cluster are no longer reporting certificate signed by unknown authority errors:
- ```
kubectl get pkgi -A
```

Additional Information

In vSphere 8.0u3 and higher, a private container registry should be used instead of manually modifying the kapp-controller configuration objects.