TMC Self Managed package reconcile issue

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After installing TMC SM you are seeing packages with below errors:

Reconcile failed: Error (see .status.usefulErrorMessage for details)   3d1htmc-local           tmc-local-stack-secrets   

Note - this KB is for TMC SM first time installation issues with packages where no clusters have been added to TMC SM. It is not for existing / establish TMC SM  issue with packages ( with existing / established TMC SM you may have quite a number of clusters managed and this KB will full uninstall TMC SM and may cause issues if you have clusters currently added to TMC SM )

Environment

TMC SM 1.4

Cause

Your values.yaml file has not been correctly configured

Resolution

Backup the values.yaml

kubectl get secret tanzu-mission-control-data values -n tmc-local -ojson | jq '.data."values.yaml"' -r | base64 -d > values.yaml.bak

Manually uninstall TMC-SM

kubectl get -n tmc-local pkgi | awk '{print $1}' | sed 1d | xargs kubectl -n tmc-local delete pkgi

Note - again, this KB is for TMC SM first time installation issues with packages where no clusters have been added to TMC SM. It is not for existing / establish TMC SM  issue with packages ( with existing / established TMC SM you may have quite a number of clusters managed and this KB will full uninstall TMC SM and may cause issues if you have clusters currently added to TMC SM )


If you find pkgi are not deleting you may need to run below commands to find what objects remain in the tmc-local namespace

kubectl -n tmc-local delete deployment --all 

kubectl -n tmc-local delete sts --all 

kubectl -n tmc-local delete pkgi --all

kubectl -n tmc-local delete postgresendpoint --all

If any issues with postgresendpoints then you may need to run below to remove the postgresendpoint finalizers

kubectl -n tmc-local get postgresendpoint | awk '{print $1}' | xargs -I{} kubectl patch postgresendpoint {} -p '{"metadata": {"finalizers":[]}}' --type merge -n tmc-local

If this does not help you may need to run below

kubectl -n tmc-local edit postgresendpoint <postgresendpoint_name> #removed finalizer one postgresendpoint at a time

Or use below to forceful delete the postgresendpoints

kubectl -n tmc-local delete postgresendpoint --all --force

Remove the tmc-local namespace

kubectl delete ns tmc-local

If you are having issue removing pkgi for tmc-local you may need to remove the finalizers manually also with below commands for example

kubectl -n tmc-local edit pkgi postgress-endpoint-controller # remove the finalizer

kubectl -n tmc-local edit pkgi tanzu-mission-control # remove the finalizer

Same may need to be done for the apps

kubectl -n tmc-local edit app postgress-endpoint-controller # remove the finalizer

kubectl -n tmc-local edit app tanzu-mission-control # remove the finalizer

Search clusterrole and clusterrolebindings for any TMC entrys with below commands

kubectl get clusterrole, clusterrolebinding -A | grep -Ei tmc

You will need to remove these if exist similar to below:

kubectl delete clusterrole tanzu-mission-control-tmc-local-cluster-role

kubectl delete clusterrole tmc-install-cluster-admin-role

kubectl delete clusterrolebinding tanzu-mission-control-tmc-local-cluster-rolerbinding

kubectl delete clusterrolebinding tmc-install-cluster-admin-role-binding

Search for any CRDs with below command

kubectl get crd -A | grep -Ei tmc

You will need to remove capabilities CRD as per below:

kubectl delete crd capabilities.secretgen-controller.tmc.tanzu.vmware.com


Next TMC SM can be reinstalled as per documents here.

Pay particular attention to values.yaml configuration and follow as per guided.

At the top of the values.yaml please make sure you add below:

kubernetesGroupVersions:
  - "secretgen.carvel.dev/v1alpha1"

Also be sure to add any required certs to the trustedCAs section example the tmc-issuer.