Extensions and Packages Reconciliation failure upon upgrading to K8s version 1.22
search cancel

Extensions and Packages Reconciliation failure upon upgrading to K8s version 1.22

book

Article ID: 327439

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Upon Upgrading TKG or TKGS Guest cluster to K8s version 1.22.x you would notice some of the packages like Contour, Cert-manager, and Harbor goes into Reconciliation Failed state.

You would see the below error when you run kubectl get app or tanzu packages installed list commands

ubuntu@jumpbox:~$ tanzu package installed list
- Retrieving installed packages... I0612 11:49:42.896265  269584 request.go:665] Waited for 1.025611064s due to client-side throttling, not priority and fairness, request: GET:https://10.225.50.9:6443/apis/ops.antrea.tanzu.vmware.com/v1alpha1?timeout=32s


  NAME    PACKAGE-NAME             PACKAGE-VERSION       STATUS
  harbor  harbor.tanzu.vmware.com  2.2.3+vmware.1-tkg.1  Reconcile failed: Error (see .status.usefulErrorMessage for details)

  usefulErrorMessage: |-
    I0612 11:40:19.136342   26373 request.go:665] Waited for 1.037915534s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.1:443/apis/batch/v1beta1?timeout=32s
    kapp: Error: Expected to find kind 'rbac.authorization.k8s.io/v1beta1/RoleBinding', but did not:
    - Kubernetes API server did not have matching apiVersion + kind
    - No matching CRD was found in given configuration

App CR status would have the below-related error in .status.usefulErrorMessage

  usefulErrorMessage: |-
    I0612 11:40:19.136342   26373 request.go:665] Waited for 1.037915534s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.1:443/apis/batch/v1beta1?timeout=32s
    kapp: Error: Expected to find kind 'rbac.authorization.k8s.io/v1beta1/RoleBinding', but did not:
    - Kubernetes API server did not have matching apiVersion + kind
    - No matching CRD was found in given configuration


Resolution

From K8s 1.22 beta has been removed from apiversion for Rolebinding https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/ . In order to fix the issue, we need to upgrade the corresponding package.

Lets take an example of Harbor package and the procedure to upgrade the same.

Note: Upgrading Harbor packages rotates the CA certificate if the certificate wss self signed and generated by cert-manager so if you are using this Harbor for Image push/pull for other clusters you would have to update this certificate to those workload clusters.

ubuntu@jumpbox:~$ tanzu package repository get tanzu-standard -n tanzu-package-repo-global
- Retrieving repository tanzu-standard... 

NAME:          tanzu-standard
VERSION:       1090396
REPOSITORY:    projects.registry.vmware.com/tkg/packages/standard/repo
TAG:           v1.4.0
STATUS:        Reconcile succeeded
REASON:
ubuntu@jumpbox:~$ tanzu package repository update tanzu-standard --url projects.registry.vmware.com/tkg/packages/standard/repo -n tanzu-package-repo-global

 Updating package repository 'tanzu-standard'
- Getting package repository 'tanzu-standard' 
 Getting package repository 'tanzu-standard'
 Validating provided settings for the package repository
 Updating package repository resource
 Waiting for 'PackageRepository' reconciliation for 'tanzu-standard'
 'PackageRepository' resource install status: Reconciling
 'PackageRepository' resource install status: ReconcileSucceeded

Updated package repository 'tanzu-standard' in namespace 'tanzu-package-repo-global'

ubuntu@jumpbox:~$ tanzu package repository list -A
- Retrieving repositories... 
  NAME            REPOSITORY                                               TAG                     STATUS               DETAILS  NAMESPACE
  tanzu-standard  projects.registry.vmware.com/tkg/packages/standard/repo  (>0.0.0)                Reconcile succeeded           tanzu-package-repo-global
  tanzu-core      projects.registry.vmware.com/tkg/packages/core/repo      v1.22.9_vmware.1-tkg.1  Reconcile succeeded           tkg-system
  • Once the tanzu-standard package repository is updated, we need to update the Harbor package. Lets get the list of available package version for Harbor
ubuntu@jumpbox:~$ tanzu package available list harbor.tanzu.vmware.com
- Retrieving package versions for harbor.tanzu.vmware.com...

  NAME                     VERSION               RELEASED-AT
  harbor.tanzu.vmware.com  2.2.3+vmware.1-tkg.1  2021-07-07 18:00:00 +0000 UTC
  harbor.tanzu.vmware.com  2.2.3+vmware.1-tkg.2  2021-07-07 18:00:00 +0000 UTC
  harbor.tanzu.vmware.com  2.3.3+vmware.1-tkg.1  2021-09-28 06:05:00 +0000 UTC
ubuntu@jumpbox:~$

As you can see the latest version of the Harbor package is 2.3.3 we can update to that version

ubuntu@jumpbox:~$ tanzu package installed update harbor -v 2.3.3+vmware.1-tkg.1 -f harbor-data-values.yaml -n default

 Updating installed package 'harbor'
- Getting package install for 'harbor' 
 Getting package install for 'harbor'
 Getting package metadata for 'harbor.tanzu.vmware.com'
 Updating secret 'harbor-default-values'
 Updating package install for 'harbor'
 Waiting for 'PackageInstall' reconciliation for 'harbor'

After the package is upgraded you would see Reconciliation success for the apps 

kubectl get apps
NAME     DESCRIPTION           SINCE-DEPLOY   AGE
harbor   Reconcile succeeded   2m5s           2d9h
ubuntu@jumpbox:~$

 

Powered by Wolken Software