PackageRepository fails to reconcile with "struct has no .annotations field or method" error
search cancel

PackageRepository fails to reconcile with "struct has no .annotations field or method" error

book

Article ID: 379203

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Management

Issue/Introduction

PackageRepository fails to reconcile as follows:

# kubectl get pkgr -A
NAMESPACE    NAME               AGE   DESCRIPTION
tkg-system   tanzu-core         72s   Reconcile failed: Deploying: Error (see .status.usefulErrorMessage for details)

# kubectl describe pkgr -n tkg-system tanzu-core
...
kapp: Error: create packagemetadata/antrea.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tkg-system:
  Applying rebase rule to packagemetadata/antrea.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tkg-system:
    Applying ytt (overlayContractV1):
      Evaluating:
- struct has no .annotations field or method (did you mean .generation?)
    in <toplevel>
      overlay.yml:93 | #@ if pkg_repo_ann in data.values.existing.metadata.annotations

Environment

TKGm 2.2

Note: this KB is only applicable to Legacy clusters, not ClusterClass-based ones.

Cause

The PackageRepository objects are failing to reconcile due to missing .metadata.annotations field in several PackageMetadata and Package objects.

This is a known kapp-controller issue (https://github.com/carvel-dev/kapp-controller/issues/1054) impacting old versions.

Resolution

Workaround

Add a .metadata.annotations field in the objects kapp-controller is targeting for the PackageRepository.

      1. Get a list of the PackageMetadata and Package objects targeted by the PackageRepository:
        # kubectl get pkgr -n tkg-system <pkgr-name> -oyaml | egrep "PackageMetadata|Package "

        E.g.:
        # kubectl get pkgr -n tkg-system tanzu-core -oyaml | egrep "PackageMetadata|Package "

        Namespace   Name                                                                     Kind             Age  Op      Op st.  Wait to  Rs  Ri
        tkg-system  antrea.tanzu.vmware.com                                                  PackageMetadata  -    create  ???     -        -   -
        ^           antrea.tanzu.vmware.com.1.9.0+vmware.2-tkg.1-advanced                    Package          -    create  ???     -        -   -
        ^           calico.tanzu.vmware.com                                                  PackageMetadata  -    create  ???     -        -   -
        ^           calico.tanzu.vmware.com.3.24.1+vmware.1-tkg.2                            Package          -    create  ???     -        -   -
        ^           kapp-controller.tanzu.vmware.com                                         PackageMetadata  -    create  ???     -        -   -
        ^           kapp-controller.tanzu.vmware.com.0.41.7+vmware.1-tkg.1                   Package          -    create  ???     -        -   -
        ^           kube-vip-cloud-provider.tanzu.vmware.com                                 PackageMetadata  -    create  ???     -        -   -
        ^           kube-vip-cloud-provider.tanzu.vmware.com.0.0.4+vmware.4-tkg.2            Package          -    create  ???     -        -   -
        ^           load-balancer-and-ingress-service.tanzu.vmware.com                       PackageMetadata  -    create  ???     -        -   -
        ^           load-balancer-and-ingress-service.tanzu.vmware.com.1.9.3+vmware.1-tkg.1  Package          -    create  ???     -        -   -
        ^           metrics-server.tanzu.vmware.com                                          PackageMetadata  -    create  ???     -        -   -
        ^           metrics-server.tanzu.vmware.com.0.6.2+vmware.1-tkg.2                     Package          -    create  ???     -        -   -
        ^           pinniped.tanzu.vmware.com                                                PackageMetadata  -    create  ???     -        -   -
        ^           pinniped.tanzu.vmware.com.0.12.1+vmware.3-tkg.4                          Package          -    create  ???     -        -   -
        ^           secretgen-controller.tanzu.vmware.com                                    PackageMetadata  -    create  ???     -        -   -
        ^           secretgen-controller.tanzu.vmware.com.0.11.2+vmware.1-tkg.3              Package          -    create  ???     -        -   -
        ^           vsphere-cpi.tanzu.vmware.com                                             PackageMetadata  -    create  ???     -        -   -
        ^           vsphere-cpi.tanzu.vmware.com.1.25.1+vmware.2-tkg.1                       Package          -    create  ???     -        -   -
        ^           vsphere-csi.tanzu.vmware.com                                             PackageMetadata  -    create  ???     -        -   -
        ^           vsphere-csi.tanzu.vmware.com.2.7.1+vmware.2-tkg.1                        Package          -    create  ???     -        -   -

      2. Create a list with the above PackageMetadata resources.

        E.g.:
        # vim pkgm_resources.txt
        antrea.tanzu.vmware.com
        calico.tanzu.vmware.com
        kapp-controller.tanzu.vmware.com
        kube-vip-cloud-provider.tanzu.vmware.com
        load-balancer-and-ingress-service.tanzu.vmware.com
        metrics-server.tanzu.vmware.com
        pinniped.tanzu.vmware.com
        secretgen-controller.tanzu.vmware.com
        vsphere-cpi.tanzu.vmware.com
        vsphere-csi.tanzu.vmware.com

      3. Add an annotation field to the above resources.

        E.g.:
        # for resource in `cat pkgm_resources.txt`; do kubectl patch pkgm -n tkg-system "$resource" --type=merge -p '{"metadata":{"annotations":{"test":"test"}}}'; done

      4. Create a list with the above Package resources.

        E.g.:
        # vim pkg_resources.txt
        antrea.tanzu.vmware.com.1.9.0+vmware.2-tkg.1-advanced
        calico.tanzu.vmware.com.3.24.1+vmware.1-tkg.2
        kapp-controller.tanzu.vmware.com.0.41.7+vmware.1-tkg.1
        kube-vip-cloud-provider.tanzu.vmware.com.0.0.4+vmware.4-tkg.2
        load-balancer-and-ingress-service.tanzu.vmware.com.1.9.3+vmware.1-tkg.1
        metrics-server.tanzu.vmware.com.0.6.2+vmware.1-tkg.2
        pinniped.tanzu.vmware.com.0.12.1+vmware.3-tkg.4
        secretgen-controller.tanzu.vmware.com.0.11.2+vmware.1-tkg.3
        vsphere-cpi.tanzu.vmware.com.1.25.1+vmware.2-tkg.1
        vsphere-csi.tanzu.vmware.com.2.7.1+vmware.2-tkg.1

      5. Add an annotation field to the above resources.

        E.g.:
        # for resource in `cat pkg_resources.txt`; do kubectl patch pkg -n tkg-system "$resource" --type=merge -p '{"metadata":{"annotations":{"test":"test"}}}'; done


Fix

The fix (https://github.com/carvel-dev/kapp-controller/pull/1127) was included in newer kapp-controller releases shipped in newer TKG versions (i.e. TKG 2.3 and onwards).