Tanzu kubernetes grid management cluster upgrade fails with kapp error if tkg-pkg package is paused

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

"tanzu mc upgrade" failed with below error message

kapp: Error: waiting on reconcile packageinstall/tanzu-addons-manager (packaging.carvel.dev/v1alpha1) namespace: tkg-system:

 Finished unsuccessfully (Reconcile failed: (message: Expected to find at least one version, but did not (details: all=1 -> after-prereleases-filter=1 -> after-kapp-controller-version-check=1 -> after-constraints-filter=0))), retrying

Environment

VMware Tanzu Kubernetes Grid 2.3.0
VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid 2.2.0
VMware Tanzu Kubernetes Grid 2.1.0
VMware Tanzu Kubernetes Grid 2.1.1

Cause

tkg-pkg would have been in paused state.

• tkg-pkg reconciles tanzu-framework
  • tanzu-framwork reconciles tanzu-addons-manager

When the tkg-pkg is paused, tanzu-framework and tanzu-addons-manager don't correctly update their information.

Resolution

Unpause tkg-pkg to reconcile the definition of tanzu-addons-manager.

Workaround:

 

Unpause "tkg-pkg"
# kubectl patch pkgi -n tkg-system tkg-pkg -p '{"spec":{"paused":false}}' --type=merge
-> tanzu-framework will be reconciled
-> tanzu-addons-manager will be reconciled

Check
# kubectl -n tkg-system get pkgi
-> tanzu-framework will be "Reconcile succeeded" 
-> tanzu-addons-manager will be "Reconcile succeeded"

If above is no problem, go with upgrade
# tanzu mc upgrade $MC_NAME -v9

 

 

 

Additional Information

Apply KB if tkr-source-controller pod hits OOM after the upgrade.