TKG Cluster stuck in deleting state in TMC due to "paused: true" variable in cluster manifest

Products

• VMware Tanzu Mission Control
• VMware Tanzu Kubernetes Grid Service (TKGs)
• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• After attempting deletion of a TKG cluster from Tanzu Mission Control, the cluster status is stuck in "Deleting" state.
• The cluster health shows "Unknown"
• VM's associated with the cluster are still present in the backing cloud provider.
• Attempting deletion from the CLI using kubectl commands also hangs and fails to delete the cluster

Environment

This might occur on TKGS or TKGm backing clusters operated by the Cluster API

Cause

If the cluster is configured with "paused: true", it will not be operated against. The Paused flag must be removed in order to allow the CAPI controllers to delete the cluster.

 

To determine if the cluster is paused, use the following command:

kubectl get cluster <CLUSTER_NAME> -n NAMESPACE_NAME -o yaml | grep -i pause

Resolution

Use the following command to remove the "paused: true" flag from the cluster to allow remediation:

kubectl patch cluster <CLUSTER_NAME> -n <NAMESPACE_NAME> -type merge -p '{"spec":{"paused": false}}'

 

Example:

kubectl patch cluster test-cluster -n test-cluster-namespace --type merge -p '{"spec":{"paused": false}}'