TKG Management cluster upgrade fails, config-values secret not found in tkg-system namespace

Products

• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:
TKG upgrade fails as it cant find the config-values secret for the Management cluster.

upgrade_region.go:230] tkg-pkg-tkg-system-values secret in tkg-system namespace not found, trying to fetch <Cluster Name>-config-values secret instead... 
clusterclient.go:1412] waiting for resource <Cluster Name>-config-values of type *v1.Secret to be up and running 
poller.go:63] secrets “<Cluster Name>-config-values” not found, retrying

Environment

2.x
3.x

Cause

The <Cluster Name>-config-values secret has been removed from tkg-system namespace.

Resolution

The following procedure can be used to restore the <Cluster Name>-config-values secret.

1. SSH to the jumpbox/TCA CP node that was used to create the Management cluster and locate cluster file used to create the cluster. It it normally in one of the following locations:
   

   ~/.config/tanzu/tkg/clusterconfigs 
   /opt/vmware/k8s-bootstrapper

2. Take a copy of the file (ex: <Cluster Name>-config.yaml) and remove references to providers, release, images and cert-manager leaving just Configuration variables, This updated file will be used to restore the secret.
   
   
3. Create the secret yaml file
   

   # kubectl create secret generic <Cluster Name>-config-values --from-file=value=cluster-config.yaml -n tkg-system --dry-run -o yaml > <Cluster Name>-config-values.yaml

4.  Update <Cluster-name>-config-values.yaml, add the following metadata at the bottom of the file, edit the two "<Cluster-name>" fields to match your clusters name
   

   metadata:
     labels:
       clusterctl.cluster.x-k8s.io/move: ""
       tkg.tanzu.vmware.com/cluster-name: <Cluster Name>
     name: <Cluster Name>-config-values
     namespace: tkg-system
   type: addons.cluster.x-k8s.io/resource-set

5. Create the secret in the Management cluster
   

   kubectl apply -f <Cluster Name>-config-values.yaml

6. Initiate Management cluster upgrade again