Customizing TMC deployed Package using overlay

Products

• VMware Tanzu Mission Control

Issue/Introduction

This article outlines the steps to create an overlay for customizing the cert-manager package deployed via TMC

Cause

If there is any customization required say adding nameservers for cert-manager to query, we can leverage flags defined in below documention. Adding them manually to cert-manager deployment will be overwritten as it is package managed by the kapp. We need to create an overlay and this article outlines those steps.
https://cert-manager.io/docs/configuration/acme/dns01/#setting-nameservers-for-dns01-self-check

Resolution

We need to create an overlay secret in the cert-manager-xxx namespace(where cert-manager pkgi is running) with the below content. Make sure secret name and yaml name under stringData is same. In our case it is "add-nameserver"

kubo@jumper:~$ cat carvel-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: add-nameserver
  namespace: cert-manager-xxx
stringData:
  add-nameserver.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind": "Deployment", "metadata": {"name": "cert-manager", "namespace": "cert-manager"}}),expects="0+"
    ---
    spec:
      template:
        spec:
          containers:
            - #@overlay/match by="name", expects=1
              name: cert-manager-controller
              #@overlay/merge
              args:
                - '--dns01-recursive-nameservers-only'
                - '--dns01-recursive-nameservers=example-dns.com:53'

Annotate the pkgi with the overlay secret using the below command

kubectl annotate -n cert-manager-xxxx pkgi cert-manager ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=add-nameserver

Wait for the package to reconcile successfully

kubectl get pkgi,app -n cert-manager-xxx

You can verify cert-manager deployment will have flags now

kubectl get pkgi,app -n cert-manager-e43d1d50
NAME                                               PACKAGE NAME                    PACKAGE VERSION          DESCRIPTION           AGE
packageinstall.packaging.carvel.dev/cert-manager   cert-manager.tanzu.vmware.com   1.12.10+vmware.1-tkg.1   Reconcile succeeded   19d

NAME                                DESCRIPTION           SINCE-DEPLOY   AGE
app.kappctrl.k14s.io/cert-manager   Reconcile succeeded   6m50s          19d

kubectl get deploy -n cert-manager
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           19d

kubectl get deploy cert-manager -n cert-manager -o yaml | grep -A7 args
      - args:
        - --v=2
        - --cluster-resource-namespace=$(POD_NAMESPACE)
        - --leader-election-namespace=kube-system
        - --acme-http01-solver-image=$(CERT_MANAGER_ACMESOLVER_IMAGE)
        - --max-concurrent-challenges=60
        - --dns01-recursive-nameservers-only
        - --dns01-recursive-nameservers=example-dns.com:53

Additional Information

For more details you can review the carvel documentation https://carvel.dev/kapp-controller/docs/v0.50.x/package-install-extensions/#adding-paths-to-ytt-overlays