TKG Management cluster upgrade fails, config-values secret not found in tkg-system namespace

Products

• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:
TKG upgrade fails as it cant find the config-values secret for the Management cluster.

upgrade_region.go:230] tkg-pkg-tkg-system-values secret in tkg-system namespace not found, trying to fetch <Cluster Name>-config-values secret instead... 
clusterclient.go:1412] waiting for resource <Cluster Name>-config-values of type *v1.Secret to be up and running 
poller.go:63] secrets “<Cluster Name>-config-values” not found, retrying

Cause

The <Cluster Name>-config-values secret has been removed from tkg-system namespace.

Resolution

The following procedure can be used to restore the <Cluster Name>-config-values secret.

• SSH to the jumpbox/TCA CP node that was used to create the Management cluster and locate cluster file used to create the cluster. It it normally in ~/.config/tanzu/tkg/clusterconfigs

 

• Take a copy of the file (eg cluster-config.yaml) and remove references to providers, release, images and cert-manager leaving just Configuration variables as described here 

 

• Create the secret yaml file 

kubectl create secret generic <Cluster-name>-config-values --from-file=value=cluster-config.yaml -n tkg-system --dry-run -o yaml > <Cluster-Name>-config-values.yaml

 

• Update <Cluster-name>-config-values.yaml, add the following metadata and type

metadata:
  labels:
    clusterctl.cluster.x-k8s.io/move: ""
    tkg.tanzu.vmware.com/cluster-name: <Cluster Name>
  name: <Cluster Name>-config-values
  namespace: tkg-system
type: addons.cluster.x-k8s.io/resource-set

 

• Create the secret in the Management cluster

kubectl config use-context <Mgmt Cluster Context>
kubectl apply -f <Cluster Name>-config-values.yaml

• Initiate Management cluster upgrade again