Pinniped Package will be in reconcile failed state with “Error : tkg_cluster_role must be provided to be either 'management' or 'workload'" in TKGm version 2.x

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid

Issue/Introduction

• The documentation for generating a pinniped add-on secret in TKGm 2.x is available only for cluster-class based management clusters.
  
  
• If users enable pinniped on legacy management clusters according to official docs mentioned above, they could hit the following errors.
  

kubo@server:~$ kubectl get pkgi -A
    NAMESPACE    NAME                            PACKAGE NAME                                     PACKAGE VERSION                                 DESCRIPTION                                                                                      AGE
     …
    tkg-system   pinniped                           pinniped.tanzu.vmware.com                   0.12.1+vmware.2-tkg.4             Reconcile failed: Error 
    (see .status.usefulErrorMessage for details)   24m

 

kubo@server:~$ kubectl describe pkgi  pinniped  -n tkg-system  

 Template:
    Error:      Templating dir: Error (see .status.usefulErrorMessage for details)
    Exit Code:  1
    Stderr:     ytt: Error:
cannot load /helpers.star: Evaluating starlark template:
    in <toplevel>
      02-kapp-config.yaml:4 | #@ load("/helpers.star", "helpers")
    reason:
     - library.export: Exporting from library 'helpers': Evaluating starlark template:
         in <toplevel>
           4 | helpers = library.get("helpers").with_data_values(data.values, plain=True).export("helpers")
         reason:
          - cannot load values.star: Evaluating starlark template:
              in <toplevel>
                33 | load("values.star",
              reason:
               - assert.fail: fail: tkg_cluster_role must be provided to be either 'management' or 'workload'
                   in validate_pinniped
                     12 |   data.values.tkg_cluster_role in ("management", "workload") or assert.fail("tkg_cluster_role must be provided

 

kubo@server:~$  kubectl get secret -A | grep pinniped
    default                             <workload-name>-pinniped-addon                                                                        tkg.tanzu.vmware.com/addon                                    1      14m
    …
    tkg-system                          <mc-name>-pinniped-addon                                                                   tkg.tanzu.vmware.com/addon                                    1      15m
    tkg-system                          <mc-name>-pinniped-package                                                                 clusterbootstrap-secret                                       1      3m46s

 

Cause

This  is because in legacy cluster pinniped secret is named `<mc-name>-pinniped-addon` and in type `tkg.tanzu.vmware.com/addon`.

However, for class-based cluster pinniped secret is named `<mc-name>-pinniped-package` and in type `clusterbootstrap-secret`.

We will be able to see difference in the contents of `values.yaml`  present inside the secret <mc name>-pinniped-addon. 

Resolution

• List pinniped add-on secrets using:

         kubectl get secret -A | grep pinniped

kubo@server:~$ kubectl get secret -A | grep pinniped                                  
default                             <workload-name>-addon                                                                    tkg.tanzu.vmware.com/addon                                    1             10m
...
tkg-system                          pinniped-data-values                                                                        Opaque                                                        1      31m
tkg-system                          pinniped-fetch-0                                                                            kubernetes.io/dockerconfigjson                                1      31m
tkg-system                          <mc-name>-pinniped-addon                                                                  tkg.tanzu.vmware.com/addon                                    1      11m
tkg-system                          <mc-name>-pinniped-package                                                                clusterbootstrap-secret                                       1      20m

• Take backup of the secret:
  
  kubectl get secret <mc-name>-pinniped-package -n tkg-system -o yaml > <mc-name>-pinniped-package_bkp.yaml
kubectl get secret <mc-name>-pinniped-addon -n tkg-system -o yaml > <mc-name>-pinniped-addon_bkp.yaml
kubectl get secret <workload-name>-pinniped-addon -n default -o yaml > <workload-name>-pinniped-addon_bkp.yaml

• Remove finalizer `tkg.tanzu.vmware.com/addon` then delete secret for MC:
  
  kubectl edit secret <mc-name>-pinniped-package -n tkg-system
kubectl delete secret <mc-name>-pinniped-package -n tkg-system

kubectl edit secret <mc-name>-pinniped-addon -n tkg-system
kubectl delete secret <mc-name>-pinniped-addon -n tkg-system
  
  
• Remove finalizer `tkg.tanzu.vmware.com/addon` then delete secret for all workload clusters:
  
  kubectl edit secret <workload-name>-pinniped-addon
kubectl delete secret <workload-name>-pinniped-addon
  
  
• Verify pinniped package is removed:
  
  kubectl get pkgi pinniped -n tkg-system
kubectl get app pinniped -n tkg-system
kubectl get ns pinniped-concierge
kubectl get ns pinniped-supervisor
  
  
• Re-install pinniped by following below steps:

         1. Generate a Pinniped add-on secret for cluster-class cluster following the documentation:

E.g. :
IDENTITY_MANAGEMENT_TYPE=ldap FILTER_BY_ADDON_TYPE="authentication/pinniped" tanzu management-cluster create <mc-name> --dry-run -f <mc-configuration-file>.yaml > <mc-name>-manifest.yaml

         
2. Manually make a change into the Pinniped add-on secret for legacy cluster:

vim <mc-name>-manifest.yaml

• Change name to <mc-name>-pinniped-addon
• Add "#@data/values\n#@overlay/match-child-defaults missing_ok=True\n---\n" at the head of values.yaml
• Change type to tkg.tanzu.vmware.com/addon

E.g.:


apiVersion: v1
kind: Secret
metadata:
  annotations:
    tkg.tanzu.vmware.com/addon-type: authentication/pinniped
  labels:
    clusterctl.cluster.x-k8s.io/move: ""
    tkg.tanzu.vmware.com/addon-name: pinniped
    tkg.tanzu.vmware.com/cluster-name: <mc-name>
  name: <mc-name>-pinniped-addon
  namespace: tkg-system
stringData:
  values.yaml: |
    #@data/values
    #@overlay/match-child-defaults missing_ok=True
    ---
    infrastructure_provider: vsphere
    tkg_cluster_role: management
    custom_cluster_issuer: ""
    custom_tls_secret: ""
    http_proxy: ""
    https_proxy: ""
    no_proxy: ""
    identity_management_type: ldap
...
type: tkg.tanzu.vmware.com/addon 

       
3. Apply Pinniped add-on secret:
   
kubectl apply -f <mc-name>-manifest.yaml

         
4. Verify no errors are shown:

kubectl get pkgi -A
kubectl get secret -A | grep pinniped