Tanzu Kubernetes Grid (TKG) 1.3.0 extensions do not function on Tanzu Kubernetes Grid Service clusters when attached to Tanzu Mission Control (TMC)
search cancel

Tanzu Kubernetes Grid (TKG) 1.3.0 extensions do not function on Tanzu Kubernetes Grid Service clusters when attached to Tanzu Mission Control (TMC)


Article ID: 339866


Updated On:





The following are some symptoms you encounter when you face this issue:

  • Tanzu Kubernetes Grid (TKG) Extensions (Contour, Fluentbit, Prometheus, Grafana) that have been previously installed and are functioning correctly on a Tanzu Kubernetes cluster, created by using Tanzu Kubernetes Grid Service, stop working when you attach that guest cluster to Tanzu Mission Control (TMC).

  • When you attach the cluster without extensions installed to TMC, installing extensions after attaching the cluster results in an error.

  • You see output similar to the following when you run the kubectl get extensions -A command:

tanzu-system-ingress   contour 

Note: The STATE column should have a value present for a successfully deployed extension.

  • You see messages similar to the following when you run kubectl get events against the namespace where the extension is installed:

LAST SEEN   TYPE      REASON                          OBJECT              MESSAGE 3m50s       Warning   LastAppliedConfigurationError   extension/contour   Last applied configuration not found for extension


VMware Tanzu Kubernetes Grid 1.x


The version of the TMC Extension Manager (tmc-extension-manager) deployed by attaching a cluster to TMC is incompatible with TKG extensions.


TKG extensions will be decoupled from TMC Extension Manager (tmc-extension-manager) in an upcoming release.


Apply the following workaround in two use cases. 

Use Case 1 - Attaching clusters with previously installed TKG extensions that are functioning

In this workaround, you will delete the extension Custom Resource corresponding to each installed extension, attach the cluster using TMC, and then validate that each extension continues to function. 

Deleting the extension Custom Resource will not delete the associated Kubernetes objects related to the extension because there are two annotations on the extension object that will prevent tmc-extension-manager from deleting app objects:

tmc.cloud.vmware.com/managed: "false",
tmc.cloud.vmware.com/orphan-resource: "true”

Run the command in this format to delete the extension Custom Resource:

kubectl delete extension <extension-name> -n <extension-namespace>

The following are commands for deleting extension objects corresponding to each extension that is part of TKG 1.3. Only delete the ones that you previously installed.

1. kubectl delete extension contour -n tanzu-system-ingress

2. kubectl delete extension fluent-bit -n tanzu-system-logging

3. kubectl delete extension grafana -n tanzu-system-monitoring

4. kubectl delete extension prometheus -n tanzu-system-monitoring

Then attach the TKG cluster using TMC. Validate that the extension is healthy by running the following command:

kubectl get apps -A 

You should see output similar to the following. Look for the Reconcile succeeded in the DESCRIPTION column.


tanzu-system-logging fluent-bit Reconcile succeeded 80s 103s

Use Case 2 - Remediating non-functioning extensions in Tanzu Kubernetes Grid Service clusters attached to TMC

This method assumes that you have attached the cluster to TMC and followed the steps in the vSphere with Tanzu documentation to install TKG extensions. For example, the steps to install fluent-bit in this documentation: Deploy the Fluent Bit Extension

Run the command to create the App Custom Resource corresponding to each extension. This bypasses the TMC extension manager. Locate the tkg-extensions bundle and navigate to the corresponding extension before running the command, while replacing <extension-name> with the corresponding extension:

kubectl apply -f <extension-name>-extension.yaml --dry-run=client -o jsonpath={.spec.objects} | kubectl apply -f- -n $(kubectl apply -f <extension-name>-extension.yaml --dry-run=client -o jsonpath={.metadata.namespace})

The following are commands for creating App objects corresponding to each extension that is a part of the TKG 1.3 extensions bundle:

1. kubectl apply -f contour-extension.yaml --dry-run=client -o jsonpath={.spec.objects} | kubectl apply -f- -n $(kubectl apply -f contour-extension.yaml --dry-run=client -o jsonpath={.metadata.namespace})

2. kubectl apply -f fluent-bit-extension.yaml --dry-run=client -o jsonpath={.spec.objects} | kubectl apply -f- -n $(kubectl apply -f fluent-bit-extension.yaml --dry-run=client -o jsonpath={.metadata.namespace})

3. kubectl apply -f grafana-extension.yaml --dry-run=client -o jsonpath={.spec.objects} | kubectl apply -f- -n $(kubectl apply -f grafana-extension.yaml --dry-run=client -o jsonpath={.metadata.namespace})

4. kubectl apply -f prometheus-extension.yaml --dry-run=client -o jsonpath={.spec.objects} | kubectl apply -f- -n $(kubectl apply -f prometheus-extension.yaml --dry-run=client -o jsonpath={.metadata.namespace})

Additional Information

Deploying TKG Extensions to Tanzu Kubernetes Clusters