Customizing Pinniped Package using overlay

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x
• VMware Tanzu Kubernetes Grid Plus
• VMware Tanzu Kubernetes Grid Plus 1.x

Issue/Introduction

This article outlines the steps to create an overlay for customizing the pinniped package

Cause

If there is any customization required say adding an annotation for a service created as part of the pinniped package managed by the kapp we need to create an overlay and this article outlines those steps.

Resolution

In this KB as mentioned we will add an annotation to the pinniped supervisor service to add the external hostname 

We need to create an overlay secret in the pinniped-supervisor namespace with the below content 

cat overlay-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-overlay-secret
  namespace: tkg-system
stringData:
  overlays.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind": "Service", "metadata": {"name": "pinniped-supervisor", "namespace": "pinniped-supervisor"}})
    ---
    metadata:
      #@overlay/match missing_ok=True
      annotations:
        #@overlay/match missing_ok=True
        external-dns.alpha.kubernetes.io/hostname: "test"

Annotate the pkgi with the overlay secret using the below command

kubectl -n tkg-system annotate packageinstalls mgmt-slot35rp34-pinniped ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=my-overlay-secret
packageinstall.packaging.carvel.dev/mgmt-slot35rp34-pinniped annotated

Delete the existing service and force the reconciliation by pausing and unpausing the app 

For pausing and triggering the reconciliation we can use the below command

kubectl patch pkgi -n tkg-system pinniped-p '{"spec":{"paused":true}}' --type=merge

kubectl patch pkgi -n tkg-system pinniped-p '{"spec":{"paused":false}}' --type=merge

Once the service is recreated it gets the annotation we have defined.

kubectl -n pinniped-supervisor get svc
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
pinniped-supervisor       NodePort    100.71.59.85     <none>        443:31234/TCP   4s
pinniped-supervisor-api   ClusterIP   100.70.175.132   <none>        443/TCP         68m
ubuntu@jumpbox:~/pinniped-new$
ubuntu@jumpbox:~/pinniped-new$ k -n pinniped-supervisor get svc pinniped-supervisor -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: test
    kapp.k14s.io/identity: v1;pinniped-supervisor//Service/pinniped-supervisor;v1
    kapp.k14s.io/original: '{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"external-dns.alpha.kubernetes.io/hostname":"test"},"labels":{"kapp.k14s.io/app":"1720077277606455403","kapp.k14s.io/association":"v1.64e71e95e5051fb68a6a4ab92b46e39c"},"name":"pinniped-supervisor","namespace":"pinniped-supervisor"},"spec":{"ports":[{"name":"https","nodePort":31234,"port":443,"protocol":"TCP","targetPort":8443}],"selector":{"app":"pinniped-supervisor","kapp.k14s.io/app":"1720077277606455403"},"type":"NodePort"}}'
    kapp.k14s.io/original-diff-md5: 12c149b7c5e3eef2f62e64d69138463e
  creationTimestamp: "2024-07-04T08:23:08Z"
  labels:
    kapp.k14s.io/app: "1720077277606455403"
    kapp.k14s.io/association: v1.64e71e95e5051fb68a6a4ab92b46e39c
  name: pinniped-supervisor
  namespace: pinniped-supervisor
  resourceVersion: "7716308"
  uid: be2b9fa6-5a56-4c8f-9897-6282d7aed441
spec:
  clusterIP: 100.71.59.85
  clusterIPs:
  - 100.71.59.85
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    nodePort: 31234
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: pinniped-supervisor
    kapp.k14s.io/app: "1720077277606455403"
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}
ubuntu@jumpbox:~/pinniped-new$

For more details you can review the carvel documentation https://carvel.dev/kapp-controller/docs/v0.50.x/package-install-extensions/#adding-paths-to-ytt-overlays