TKG workload cluster upgrade fails with Error "no available upgrades for cluster"

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
Tanzu cluster upgrade fails with below error message

compatibility file (/home/xxx/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /home/xxx/.config/tanzu/tkg/bom already exists, skipping download
Error: no available upgrades for cluster 'WORKLOAD_CLUSTER_NAME', namespace 'NAMESPACE'

# tanzu kubernetes-release get

# There is no upgrade version TKr (example: v2.1.1 to v2.2.0)
NAME                       VERSION                  COMPATIBLE  ACTIVE  UPDATES AVAILABLE
v1.21.14---vmware.2-tkg.5  v1.21.14+vmware.2-tkg.5  True        True    True
v1.22.13---vmware.1-tkg.1  v1.22.13+vmware.1-tkg.1  True        True    True
v1.23.10---vmware.1-tkg.1  v1.23.10+vmware.1-tkg.1  True        True    False
v1.24.10---vmware.1-tkg.2  v1.24.10+vmware.1-tkg.2  True        True

Environment

VMware Tanzu Kubernetes Grid 2.2.0
VMware Tanzu Kubernetes Grid 2.3.0
VMware Tanzu Kubernetes Grid 2.1.0
VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid 2.1.1

Cause

tkg-pkg would have been in paused state.

• tkg-pkg reconciles both tanzu-framework and tkr-source-controller
• tanzu-framwork reconciles tanzu-addons-manager

When the tkg-pkg is paused, tanzu-framework and tanzu-addons-manager don't correctly update their information.

Resolution

Unpause both tkg-pkg and tkr-source-controller

Workaround:

Unpause both tkg-pkg and tkr-source-controller
# kubectl patch pkgi -n tkg-system tkg-pkg -p '{"spec":{"paused":false}}' --type=merge
# kubectl patch pkgi -n tkg-system tkr-source-controller -p '{"spec":{"paused":false}}' --type=merge

Restart the pods
# kubectl -n tkg-system rollout restart deployment.apps/kapp-controller
# kubectl -n tkg-system rollout restart deployment.apps/tanzu-addons-controller-manager

List packages installed and verify component version are upgraded to latest
# kubectl get pkgi

Target TKr (v1.25.7) will be shown when upgrading to v2.2.0
# tanzu kubernetes-release get
NAME                       VERSION                  COMPATIBLE  ACTIVE  UPDATES AVAILABLE
v1.23.17---vmware.1-tkg.1  v1.23.17+vmware.1-tkg.1  True        True    
v1.24.11---vmware.1-tkg.1  v1.24.11+vmware.1-tkg.1  True        True
v1.25.7---vmware.2-tkg.1   v1.25.7+vmware.2-tkg.1   True        True

Retry workload cluster upgrade
# tanzu cluster upgrade $WORKLOAD_CLUSTER -v9

Additional Information

Apply the following KB if tkr-source-controller pod hits OOMkilled: https://knowledge.broadcom.com/external/article?legacyId=92524