Tanzu kubernetes grid management cluster upgrade fails with kapp error if tkg-pkg package is paused

Products

• Tanzu Kubernetes Grid

Issue/Introduction

"tanzu mc upgrade" failed with below error message

kapp: Error: waiting on reconcile packageinstall/tanzu-addons-manager (packaging.carvel.dev/v1alpha1) namespace: tkg-system:

 Finished unsuccessfully (Reconcile failed: (message: Expected to find at least one version, but did not (details: all=1 -> after-prereleases-filter=1 -> after-kapp-controller-version-check=1 -> after-constraints-filter=0))), retrying

Environment

• VMware Tanzu Kubernetes Grid 2.3.0
• VMware Tanzu Kubernetes Grid 2.2.0
• VMware Tanzu Kubernetes Grid 2.1.1
• VMware Tanzu Kubernetes Grid 2.1.0
• VMware Tanzu Kubernetes Grid 1.x

Cause

tkg-pkg would have been in paused state.

• tkg-pkg reconciles tanzu-framework
  • tanzu-framwork reconciles tanzu-addons-manager

When the tkg-pkg is paused, tanzu-framework and tanzu-addons-manager don't correctly update their information.

Resolution

Unpause tkg-pkg to reconcile the definition of tanzu-addons-manager.

1. Unpause "tkg-pkg"

kubectl patch pkgi -n tkg-system tkg-pkg -p '{"spec":{"paused":false}}' --type=merge
# --> tanzu-framework will be reconciled
# --> tanzu-addons-manager will be reconciled

2. Check

kubectl -n tkg-system get pkgi
# --> tanzu-framework will be "Reconcile succeeded" 
# --> tanzu-addons-manager will be "Reconcile succeeded"

3. Go to upgrade

tanzu mc upgrade $MC_NAME -v9

Additional Information

If tkr-source-controller pod hits OOM after the upgrade, Apply KB - How to Increase tkr-source-controller deployment resource limits on TKG 2.1.x to resolve 'OOMKilled' Error.