kapp-controller reconciliation fails when private registry uses self-signed certs or when registry is behind a proxy

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

During the installation of contour as part of the TKG v1.2 extensions, the contour app fails to deploy/reconcile as the kapp-controller Pod is not able to pull the images from the private registry. kapp-controller uses imgpkg to pull images from the repository. 

kapp-controller fails with the following error:

kubectl get app contour -n tanzu-system-ingress
NAME DESCRIPTION SINCE-DEPLOY AGE
contour Reconcile failed: Fetching (0): Fetching registry image: Imgpkg: exit status 1 (stderr: Error: Collecting images: Working with <PRIVATE-REGISTRY-FQDN:PORT>/tkg-extensions-templates:v1.2.0_vmware.1: Get https://<PRIVATE-REGISTRY-FQDN:PORT>: x509: certificate signed by unknown authority

Run imgpkg from within the Pod to check if it can pull the image from the private registry:

kubectl exec -n vmware-system-tmc KAPP_CONTROLLER_POD -- imgpkg pull -i PRIVATE_REGISTRY:PORT/IMAGE_NAME -o /tmp/filename --registry-ca-cert-path=/etc/ssl/certs/ca-certificates.crt​​​​​

Environment

Product Version: 1.0

Resolution

The kapp-controller pod is expecting the CA cert for the private registry to be in /etc/ssl/certs/ca-certificates.crt. As its a self-singed cert, it will not be present and must be manually added. This KB describes how to add it using a configMap.

Retrieve the ca-certificates.crt file from kapp-controller image:

id=$(docker create registry.tkg.vmware.run/kapp-controller:v0.9.0_vmware)
docker cp $id:/etc/ssl/certs/ca-certificates.crt ca-certificates.crt
docker rm $id

Append the private registry's self-signed cert to ca-certificates.crt

cat self-signed.crt >> ca-certificates.crt

Check that imgpkg is able to pull the image at Worker level, this also verifies that ca-certificates.crt has valid self-signed cert. imgpkg is part of the TKG CLI bundle.

imgpkg pull -i PRIVATE_REGISTRY:PORT/IMAGE_NAME -o /tmp/filename --registry-ca-cert-path=ca-certificates.crt

If the image was successfully pulled from the Worker, then mount the ca-certificates.crt to the kapp-controller pod.

Create a config map:

kubectl create configmap ca-certificates -n vmware-system-tmc --from-file=ca-certificates.crt

Mount ca-certificates.crt configmap to kapp-controller pod under path /etc/ssl/certs/ca-certificates.crt.

Change the following in kapp-controller Deployment in kapp-controller.yaml, data in bold should be added:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kapp-controller
  namespace: vmware-system-tmc
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: kapp-controller
  template:
    metadata:
      labels:
        app: kapp-controller
    spec:
      containers:
      - args: []
        command:
        - /kapp-controller
        env:
        - name: KAPPCTRL_MEM_TMP_DIR
          value: /etc/kappctrl-mem-tmp
        image: registry.tkg.vmware.run/kapp-controller:v0.9.0_vmware.1
        imagePullPolicy: IfNotPresent
        name: kapp-controller
        resources:
          requests:
            cpu: 120m
            memory: 100Mi
        securityContext:
          runAsGroup: 2000
          runAsUser: 1000
        volumeMounts:
        - mountPath: /etc/kappctrl-mem-tmp
          name: template-fs
        - mountPath: /etc/ssl/certs/ca-certificates.crt
          subPath: ca-certificates.crt
          name: ca-certificates
          readOnly: true
      securityContext:
        fsGroup: 3000
      serviceAccount: kapp-controller-sa
      volumes:
      - emptyDir:
          medium: Memory
        name: template-fs
      - configMap:
          name: ca-certificates
        name: ca-certificates

Apply modified kapp-controller.yaml:

kubectl apply -f kapp-controller.yaml

Check contour app:

kubectl get app contour -n tanzu-system-ingress 

If the image registry is accessible through an HTTP/HTTPS proxy, then add http, http(s), and no_proxy environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kapp-controller
  namespace: vmware-system-tmc
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: kapp-controller
  template:
    metadata:
      labels:
        app: kapp-controller
    spec:
      containers:
      - args: []
        command:
        - /kapp-controller
        env:
        - name: KAPPCTRL_MEM_TMP_DIR
          value: /etc/kappctrl-mem-tmp
        - name: http_proxy
          value: http://<proxy_host>:<proxy_port>
        - name: https_proxy
          value: http://<proxy_host>:<proxy_port>
        - name: no_proxy
          value: localhost,127.0.0.1,kubernetes.default.svc,.svc
        - name: HTTP_PROXY
          value: http://<proxy_host>:<proxy_port>
        - name: HTTPS_PROXY
          value: http://<proxy_host>:<proxy_port>       
        - name: NO_PROXY
          value:  localhost,127.0.0.1,kubernetes.default.svc,.svc       
        - name: PROXY_ENABLED
          value: "yes”
        image: registry.tkg.vmware.run/kapp-controller:v0.9.0_vmware.1
        imagePullPolicy: IfNotPresent
        name: kapp-controller
        resources:
          requests:
            cpu: 120m
            memory: 100Mi
        securityContext:
          runAsGroup: 2000
          runAsUser: 1000
        volumeMounts:
        - mountPath: /etc/kappctrl-mem-tmp
          name: template-fs
      securityContext:
        fsGroup: 3000
      serviceAccount: kapp-controller-sa
      volumes:
      - emptyDir:
          medium: Memory
        name: template-fs

Apply modified kapp-controller.yaml:

kubectl apply -f kapp-controller.yaml