TKG workload cluster upgrade fails with Error "no available upgrades for cluster"

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Tanzu cluster upgrade fails with below error message

compatibility file (/home/xxx/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /home/xxx/.config/tanzu/tkg/bom already exists, skipping download
Error: no available upgrades for cluster 'WORKLOAD_CLUSTER_NAME', namespace 'NAMESPACE'

# tanzu kubernetes-release get

# There is no upgrade version TKr (example: v2.1.1 to v2.2.0)
NAME                       VERSION                  COMPATIBLE  ACTIVE  UPDATES AVAILABLE
v1.21.14---vmware.2-tkg.5  v1.21.14+vmware.2-tkg.5  True        True    True
v1.22.13---vmware.1-tkg.1  v1.22.13+vmware.1-tkg.1  True        True    True
v1.23.10---vmware.1-tkg.1  v1.23.10+vmware.1-tkg.1  True        True    False
v1.24.10---vmware.1-tkg.2  v1.24.10+vmware.1-tkg.2  True        True

Environment

• VMware Tanzu Kubernetes Grid 2.2.0
• VMware Tanzu Kubernetes Grid 2.3.0
• VMware Tanzu Kubernetes Grid 2.1.1
• VMware Tanzu Kubernetes Grid 2.1.0
• VMware Tanzu Kubernetes Grid 1.x

Cause

tkg-pkg would have been in paused state.

• tkg-pkg reconciles both tanzu-framework and tkr-source-controller
• tanzu-framwork reconciles tanzu-addons-manager

When the tkg-pkg is paused, tanzu-framework and tanzu-addons-manager don't correctly update their information.

Resolution

Unpause both tkg-pkg and tkr-source-controller

 

 

1. Unpause both tkg-pkg and tkr-source-controller 

kubectl patch pkgi -n tkg-system tkg-pkg -p '{"spec":{"paused":false}}' --type=merge
kubectl patch pkgi -n tkg-system tkr-source-controller -p '{"spec":{"paused":false}}' --type=merge

 

2. Restart the pods

kubectl -n tkg-system rollout restart deployment.apps/kapp-controller
kubectl -n tkg-system rollout restart deployment.apps/tanzu-addons-controller-manager

 

3. List packages installed and verify component version is upgraded to the latest

Target TKr (v1.25.7) will be shown when upgrading to v2.2.0

tanzu kubernetes-release get
#> NAME VERSION COMPATIBLE ACTIVE UPDATES AVAILABLE
#> v1.23.17---vmware.1-tkg.1 v1.23.17+vmware.1-tkg.1 True True
#> v1.24.11---vmware.1-tkg.1 v1.24.11+vmware.1-tkg.1 True True
#> v1.25.7---vmware.2-tkg.1 v1.25.7+vmware.2-tkg.1 True True

 

4. Retry workload cluster upgrade

 tanzu cluster upgrade ${WORKLOAD_CLUSTER} -v9

Additional Information

How to Increase tkr-source-controller deployment resource limits on TKG 2.1.x to resolve 'OOMKilled' Error